[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:24:33 dist-test-slave-xkrt krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:24:33 dist-test-slave-xkrt krb5kdc[399](info): set up 2 sockets
Jun 19 02:24:33 dist-test-slave-xkrt krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 02:24:34.945771 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:24:35 dist-test-slave-xkrt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:24:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:24:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:24:35.066267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34367
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:35.188481 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:35.188750 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:35.188797 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:35.192874 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:24:35.192931 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:35.192950 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:35.192967 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:24:35.192982 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:24:35.198158 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:35.199138 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:35.200107 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:24:35.209340 412 server_base.cc:1034] running on GCE node
W20260619 02:24:35.209375 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:35.209880 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:35.209928 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:35.210505 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:35.211658 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:35.212853 412 hybrid_clock.cc:648] HybridClock initialized: now 1781835875212820 us; error 46 us; skew 500 ppm
Jun 19 02:24:35 dist-test-slave-xkrt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:35.215541 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:24:35.216341 412 webserver.cc:460] Webserver started at http://127.0.98.62:45567/ using document root <none> and password file <none>
I20260619 02:24:35.216848 412 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:35.216903 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:35.217058 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:35.219158 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "2b74a02f8ff847e3b655e30c4edf5b0f"
format_stamp: "Formatted at 2026-06-19 02:24:35 on dist-test-slave-xkrt"
server_key: "ce2881eda64131fee44292d0a3117256"
server_key_iv: "82e152fa0d37ceac9494ed70f704ad6d"
server_key_version: "encryptionkey@0"
I20260619 02:24:35.219671 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b74a02f8ff847e3b655e30c4edf5b0f"
format_stamp: "Formatted at 2026-06-19 02:24:35 on dist-test-slave-xkrt"
server_key: "ce2881eda64131fee44292d0a3117256"
server_key_iv: "82e152fa0d37ceac9494ed70f704ad6d"
server_key_version: "encryptionkey@0"
I20260619 02:24:35.223356 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:24:35.225706 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:35.226881 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:24:35.226984 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "2b74a02f8ff847e3b655e30c4edf5b0f"
format_stamp: "Formatted at 2026-06-19 02:24:35 on dist-test-slave-xkrt"
server_key: "ce2881eda64131fee44292d0a3117256"
server_key_iv: "82e152fa0d37ceac9494ed70f704ad6d"
server_key_version: "encryptionkey@0"
I20260619 02:24:35.227068 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:35.242427 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:35.245632 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:35.245803 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:35.252560 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34367
I20260619 02:24:35.252574 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34367 every 8 connection(s)
I20260619 02:24:35.253516 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:24:35.256604 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:35.261907 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f: Bootstrap starting.
I20260619 02:24:35.262574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 412
I20260619 02:24:35.262740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:24:35.264313 392 external_mini_cluster.cc:1428] Setting key e402abc78c6b1bd4ce68b8fa893b587c
I20260619 02:24:35.264943 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:35.265863 480 log.cc:824] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:35.267952 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f: No bootstrap required, opened a new log
I20260619 02:24:35.271860 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b74a02f8ff847e3b655e30c4edf5b0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34367 } }
I20260619 02:24:35.272123 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:35.272181 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b74a02f8ff847e3b655e30c4edf5b0f, State: Initialized, Role: FOLLOWER
I20260619 02:24:35.272655 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b74a02f8ff847e3b655e30c4edf5b0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34367 } }
I20260619 02:24:35.272823 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:24:35.272892 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:24:35.272984 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:35.273960 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b74a02f8ff847e3b655e30c4edf5b0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34367 } }
I20260619 02:24:35.274291 480 leader_election.cc:302] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b74a02f8ff847e3b655e30c4edf5b0f; no voters:
I20260619 02:24:35.274605 480 leader_election.cc:288] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:24:35.274681 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:35.274847 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [term 1 LEADER]: Becoming Leader. State: Replica: 2b74a02f8ff847e3b655e30c4edf5b0f, State: Running, Role: LEADER
I20260619 02:24:35.275127 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b74a02f8ff847e3b655e30c4edf5b0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34367 } }
I20260619 02:24:35.275833 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:24:35.276895 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b74a02f8ff847e3b655e30c4edf5b0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b74a02f8ff847e3b655e30c4edf5b0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34367 } } }
I20260619 02:24:35.276901 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b74a02f8ff847e3b655e30c4edf5b0f. Latest consensus state: current_term: 1 leader_uuid: "2b74a02f8ff847e3b655e30c4edf5b0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b74a02f8ff847e3b655e30c4edf5b0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34367 } } }
I20260619 02:24:35.277042 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:35.277050 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:35.277429 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:24:35.280740 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 02:24:35 dist-test-slave-xkrt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:35.287068 494 catalog_manager.cc:1347] Generated new cluster ID: 18a5a70a3f0e43b89ed9e7f7b9d00a56
I20260619 02:24:35.287134 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:24:35.291304 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:35.267628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42680 (local address 127.0.98.62:34367)
0619 02:24:35.268148 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:35.268167 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:35.269337 (+ 1170us) server_negotiation.cc:407] Connection header received
0619 02:24:35.272003 (+ 2666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:35.272020 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:35.272411 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:35.272774 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:35.275588 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:35.277066 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:35.278871 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:35.279596 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:35.279758 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:35.284684 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:35.284691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:35.284706 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:35.284736 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:35.287225 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:35.287853 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:35.287859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:35.287864 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:35.288043 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:35.288394 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:35.288398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:35.288399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:35.288840 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:35.289017 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:35.289788 (+ 771us) server_negotiation.cc:299] Negotiation successful
0619 02:24:35.290081 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":183,"threads_started":1}
I20260619 02:24:35.306339 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:24:35.307144 494 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:24:35.314435 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b74a02f8ff847e3b655e30c4edf5b0f: Generated new TSK 0
I20260619 02:24:35.315153 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:24:35.349012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34367
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:35.457551 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:35.457831 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:35.457907 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:35.462212 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:35.462292 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:35.462381 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:24:35.467506 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34367
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:35.468598 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:35.469651 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:35.479084 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:35.479126 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:35.479116 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:35.479108 506 server_base.cc:1034] running on GCE node
I20260619 02:24:35.479753 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:35.480347 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:35.481513 506 hybrid_clock.cc:648] HybridClock initialized: now 1781835875481500 us; error 23 us; skew 500 ppm
Jun 19 02:24:35 dist-test-slave-xkrt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:35.484329 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:24:35.485142 506 webserver.cc:460] Webserver started at http://127.0.98.1:36769/ using document root <none> and password file <none>
I20260619 02:24:35.485749 506 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:35.485838 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:35.486057 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:35.488001 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "6834ef8dde6c4225a4d8d5efceefc83f"
format_stamp: "Formatted at 2026-06-19 02:24:35 on dist-test-slave-xkrt"
server_key: "a197b66ec3ec1113b604485b75af14f9"
server_key_iv: "41ceff900fed4382528206e13d8d3cc1"
server_key_version: "encryptionkey@0"
I20260619 02:24:35.488583 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6834ef8dde6c4225a4d8d5efceefc83f"
format_stamp: "Formatted at 2026-06-19 02:24:35 on dist-test-slave-xkrt"
server_key: "a197b66ec3ec1113b604485b75af14f9"
server_key_iv: "41ceff900fed4382528206e13d8d3cc1"
server_key_version: "encryptionkey@0"
I20260619 02:24:35.492111 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:24:35.494493 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:35.495604 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:24:35.495736 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "6834ef8dde6c4225a4d8d5efceefc83f"
format_stamp: "Formatted at 2026-06-19 02:24:35 on dist-test-slave-xkrt"
server_key: "a197b66ec3ec1113b604485b75af14f9"
server_key_iv: "41ceff900fed4382528206e13d8d3cc1"
server_key_version: "encryptionkey@0"
I20260619 02:24:35.495843 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:35.519671 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:35.522967 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:35.523176 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:35.523742 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:35.524547 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:35.524621 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:35.524683 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:35.524736 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:35.534129 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37449
I20260619 02:24:35.534154 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37449 every 8 connection(s)
I20260619 02:24:35.535030 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:24:35 dist-test-slave-xkrt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:35.545168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 506
I20260619 02:24:35.545285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:24:35.545620 392 external_mini_cluster.cc:1428] Setting key 8bbd9c44e9c63b399c2e62715f853ed3
I20260619 02:24:35.548286 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:35.536758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42961 (local address 127.0.98.62:34367)
0619 02:24:35.536932 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:35.536937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:35.537779 (+ 842us) server_negotiation.cc:407] Connection header received
0619 02:24:35.539054 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:35.539057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:35.539102 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:35.539183 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:35.540292 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:35.540908 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:35.542067 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:35.542298 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:35.542381 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:35.544651 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:35.544654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:35.544657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:35.544679 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:35.546294 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:35.546769 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:35.546772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:35.546774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:35.546858 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:35.547248 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:35.547254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:35.547257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:35.547434 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:35.547536 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:35.548001 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 02:24:35.548121 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 02:24:35.549155 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:35.537155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34367 (local address 127.0.98.1:42961)
0619 02:24:35.537630 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:35.537681 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:35.538853 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:35.539331 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:35.539338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:35.539688 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:35.540162 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:35.540175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:35.541017 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:35.541020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:35.541957 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:35.541966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:35.542426 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:35.542428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:35.542558 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:35.543067 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:35.543081 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:35.544484 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:35.546383 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:35.546388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:35.546401 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:35.546670 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:35.546971 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:35.546974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:35.546976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:35.547128 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:35.547546 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:35.547553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:35.547777 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 02:24:35.547960 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 02:24:35.548265 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":108,"threads_started":1}
I20260619 02:24:35.550307 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34367
I20260619 02:24:35.550554 635 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:35.551149 635 heartbeater.cc:511] Master 127.0.98.62:34367 requested a full tablet report, sending...
I20260619 02:24:35.552650 438 ts_manager.cc:194] Registered new tserver with Master: 6834ef8dde6c4225a4d8d5efceefc83f (127.0.98.1:37449)
I20260619 02:24:35.553812 438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42961
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:24:35.571993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34367
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:35.679282 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:35.679580 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:35.679653 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:35.683615 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:35.683701 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:35.683790 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:24:35.688861 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34367
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:35.689859 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:35.690840 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:35.701529 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:35.701529 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:35.701706 640 server_base.cc:1034] running on GCE node
W20260619 02:24:35.701820 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:35.702193 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:35.702818 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:35.704007 640 hybrid_clock.cc:648] HybridClock initialized: now 1781835875703978 us; error 54 us; skew 500 ppm
Jun 19 02:24:35 dist-test-slave-xkrt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:35.706964 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:24:35.707803 640 webserver.cc:460] Webserver started at http://127.0.98.2:33573/ using document root <none> and password file <none>
I20260619 02:24:35.708336 640 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:35.708434 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:35.708663 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:35.710825 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "aa200e2f55a440e58db732d5aae963ae"
format_stamp: "Formatted at 2026-06-19 02:24:35 on dist-test-slave-xkrt"
server_key: "27f0f867692227f5662a0a958d2954ae"
server_key_iv: "99de3ab2d39a4c3c872ff7e3024903a1"
server_key_version: "encryptionkey@0"
I20260619 02:24:35.711385 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "aa200e2f55a440e58db732d5aae963ae"
format_stamp: "Formatted at 2026-06-19 02:24:35 on dist-test-slave-xkrt"
server_key: "27f0f867692227f5662a0a958d2954ae"
server_key_iv: "99de3ab2d39a4c3c872ff7e3024903a1"
server_key_version: "encryptionkey@0"
I20260619 02:24:35.714962 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:24:35.717234 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:35.718446 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:24:35.718537 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "aa200e2f55a440e58db732d5aae963ae"
format_stamp: "Formatted at 2026-06-19 02:24:35 on dist-test-slave-xkrt"
server_key: "27f0f867692227f5662a0a958d2954ae"
server_key_iv: "99de3ab2d39a4c3c872ff7e3024903a1"
server_key_version: "encryptionkey@0"
I20260619 02:24:35.718690 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:35.742813 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:35.745829 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:35.746033 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:35.746587 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:35.747392 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:35.747439 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:35.747509 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:35.747543 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:35.757941 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45775
I20260619 02:24:35.758013 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45775 every 8 connection(s)
I20260619 02:24:35.758853 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:24:35.767822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 640
I20260619 02:24:35.767973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
Jun 19 02:24:35 dist-test-slave-xkrt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:35.768291 392 external_mini_cluster.cc:1428] Setting key 0ddad24d43080ddf4c0020bfa7037e84
I20260619 02:24:35.772855 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:35.760696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33779 (local address 127.0.98.62:34367)
0619 02:24:35.760828 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:35.760832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:35.761724 (+ 892us) server_negotiation.cc:407] Connection header received
0619 02:24:35.762643 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:35.762646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:35.762704 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:35.762813 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:35.763989 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:35.764516 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:35.765726 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:35.765952 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:35.766065 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:35.768755 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:35.768758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:35.768760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:35.768789 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:35.770777 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:35.771277 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:35.771284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:35.771289 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:35.771389 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:35.771724 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:35.771728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:35.771730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:35.771929 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:35.772039 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:35.772561 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 02:24:35.772689 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:24:35.773722 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:35.761100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34367 (local address 127.0.98.2:33779)
0619 02:24:35.761527 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:35.761619 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:35.762437 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:35.762989 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:35.762996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:35.763373 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:35.763842 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:35.763855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:35.764641 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:35.764645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:35.765535 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:35.765607 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:35.766101 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:35.766105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:35.766286 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:35.766915 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:35.766932 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:35.768549 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:35.770906 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:35.770912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:35.770924 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:35.771178 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:35.771483 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:35.771485 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:35.771487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:35.771613 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:35.772036 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:35.772043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:35.772297 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 02:24:35.772505 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 02:24:35.772829 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":115,"threads_started":1}
I20260619 02:24:35.774853 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34367
I20260619 02:24:35.775110 769 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:35.775593 769 heartbeater.cc:511] Master 127.0.98.62:34367 requested a full tablet report, sending...
I20260619 02:24:35.776683 438 ts_manager.cc:194] Registered new tserver with Master: aa200e2f55a440e58db732d5aae963ae (127.0.98.2:45775)
I20260619 02:24:35.777297 438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33779
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:24:35.794699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34367
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:35.903038 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:35.903337 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:35.903432 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:35.907459 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:35.907541 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:35.907624 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:24:35.912883 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34367
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:35.913987 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:35.914994 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:35.924671 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:35.924660 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:35.924664 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:35.924988 774 server_base.cc:1034] running on GCE node
I20260619 02:24:35.925426 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:35.926045 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:35.927230 774 hybrid_clock.cc:648] HybridClock initialized: now 1781835875927215 us; error 24 us; skew 500 ppm
Jun 19 02:24:35 dist-test-slave-xkrt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:35.929852 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:24:35.930757 774 webserver.cc:460] Webserver started at http://127.0.98.3:45239/ using document root <none> and password file <none>
I20260619 02:24:35.931373 774 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:35.931447 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:35.931679 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:35.933706 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "d8dea9c09e2d4bcb880c041f3ad6fb48"
format_stamp: "Formatted at 2026-06-19 02:24:35 on dist-test-slave-xkrt"
server_key: "951bad44f1b6baf49d9102de3cf54651"
server_key_iv: "2f7a63025a437d10fa46879585cb26f2"
server_key_version: "encryptionkey@0"
I20260619 02:24:35.934267 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d8dea9c09e2d4bcb880c041f3ad6fb48"
format_stamp: "Formatted at 2026-06-19 02:24:35 on dist-test-slave-xkrt"
server_key: "951bad44f1b6baf49d9102de3cf54651"
server_key_iv: "2f7a63025a437d10fa46879585cb26f2"
server_key_version: "encryptionkey@0"
I20260619 02:24:35.937924 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:24:35.940213 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:35.941452 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:24:35.941617 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "d8dea9c09e2d4bcb880c041f3ad6fb48"
format_stamp: "Formatted at 2026-06-19 02:24:35 on dist-test-slave-xkrt"
server_key: "951bad44f1b6baf49d9102de3cf54651"
server_key_iv: "2f7a63025a437d10fa46879585cb26f2"
server_key_version: "encryptionkey@0"
I20260619 02:24:35.941733 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:35.963017 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:35.965925 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:35.966132 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:35.966718 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:35.967543 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:35.967614 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:35.967686 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:35.967730 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:35.977213 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46609
I20260619 02:24:35.977238 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46609 every 8 connection(s)
I20260619 02:24:35.978206 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:24:35.980285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 774
I20260619 02:24:35.980401 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:24:35.980664 392 external_mini_cluster.cc:1428] Setting key bf31876edb9c90deb7bb28f416df6c7b
Jun 19 02:24:35 dist-test-slave-xkrt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:35.991794 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:35.979995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46205 (local address 127.0.98.62:34367)
0619 02:24:35.980172 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:35.980179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:35.981102 (+ 923us) server_negotiation.cc:407] Connection header received
0619 02:24:35.982593 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:35.982596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:35.982652 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:35.982738 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:35.983901 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:35.984418 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:35.985485 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:35.985767 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:35.985850 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:35.988526 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:35.988530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:35.988532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:35.988564 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:35.989841 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:35.990343 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:35.990348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:35.990350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:35.990437 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:35.990768 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:35.990773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:35.990775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:35.990949 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:35.991054 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:35.991522 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 02:24:35.991636 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 02:24:35.992599 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:35.980450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34367 (local address 127.0.98.3:46205)
0619 02:24:35.980928 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:35.980980 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:35.982398 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:35.982940 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:35.982947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:35.983312 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:35.983776 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:35.983790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:35.984517 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:35.984519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:35.985379 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:35.985387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:35.985890 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:35.985893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:35.986034 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:35.986951 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:35.986966 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:35.988357 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:35.989959 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:35.989965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:35.989979 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:35.990238 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:35.990531 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:35.990536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:35.990538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:35.990672 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:35.991050 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:35.991056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:35.991295 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 02:24:35.991481 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 02:24:35.991790 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":129,"threads_started":1}
I20260619 02:24:35.993788 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34367
I20260619 02:24:35.994029 903 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:35.994504 903 heartbeater.cc:511] Master 127.0.98.62:34367 requested a full tablet report, sending...
I20260619 02:24:35.995450 438 ts_manager.cc:194] Registered new tserver with Master: d8dea9c09e2d4bcb880c041f3ad6fb48 (127.0.98.3:46609)
I20260619 02:24:35.996019 438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46205
I20260619 02:24:36.006565 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:24:36 dist-test-slave-xkrt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:24:36 dist-test-slave-xkrt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 02:24:36.031823 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:36.022659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.0.98.1:37449)
0619 02:24:36.022979 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:36.022993 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:36.023016 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:24:36.023068 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:36.023073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:36.023313 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:36.023442 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:36.024021 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:36.024828 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:36.025665 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:36.026049 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:36.026132 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:36.028476 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:36.028485 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:36.028497 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:36.028520 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:36.030075 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:36.030540 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:36.030545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:36.030546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:36.030625 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:36.030858 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:36.030860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:36.030861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:36.031088 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:36.031251 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:36.031439 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 02:24:36.031637 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":115,"threads_started":1}
W20260619 02:24:36.032771 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43608
Jun 19 02:24:36 dist-test-slave-xkrt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:24:36 dist-test-slave-xkrt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 02:24:36.057772 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:36.049795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.0.98.1:37449)
0619 02:24:36.049926 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:36.049930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:36.050004 (+ 74us) server_negotiation.cc:407] Connection header received
0619 02:24:36.050180 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:36.050184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:36.050238 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:36.050325 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:36.051041 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:36.051562 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:36.052402 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:36.052655 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:36.052767 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:36.055136 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:36.055139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:36.055141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:36.055162 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:36.056401 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:36.056755 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:36.056758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:36.056759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:36.056827 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:36.057043 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:36.057045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:36.057046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:36.057193 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:36.057286 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:36.057476 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 02:24:36.057635 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:24:36.058866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 506
I20260619 02:24:36.065251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 640
I20260619 02:24:36.072661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 774
I20260619 02:24:36.077942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 412
2026-06-19T02:24:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2188 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:24:36 dist-test-slave-xkrt krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:24:36 dist-test-slave-xkrt krb5kdc[917](info): set up 2 sockets
Jun 19 02:24:36 dist-test-slave-xkrt krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 02:24:37.109159 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:24:37 dist-test-slave-xkrt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:24:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:24:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:24:37.228735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44819
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:37.335227 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:37.335502 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:37.335580 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:37.339900 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:24:37.339996 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:37.340047 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:37.340106 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:24:37.340142 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:24:37.345474 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:37.346503 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:37.347478 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:37.355815 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:37.355832 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:37.355881 930 server_base.cc:1034] running on GCE node
W20260619 02:24:37.355800 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:37.356554 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:37.357642 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:37.358835 930 hybrid_clock.cc:648] HybridClock initialized: now 1781835877358825 us; error 31 us; skew 500 ppm
Jun 19 02:24:37 dist-test-slave-xkrt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:37.365094 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:24:37.365957 930 webserver.cc:460] Webserver started at http://127.0.98.62:33049/ using document root <none> and password file <none>
I20260619 02:24:37.366593 930 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:37.366653 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:37.366822 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:37.368721 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "1033e8b8bd66449291282189e3a99d8d"
format_stamp: "Formatted at 2026-06-19 02:24:37 on dist-test-slave-xkrt"
server_key: "fae3ed0ff4b10e79d225662357f264e1"
server_key_iv: "c43a7f43b3ab899738a46d42c338338e"
server_key_version: "encryptionkey@0"
I20260619 02:24:37.369438 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "1033e8b8bd66449291282189e3a99d8d"
format_stamp: "Formatted at 2026-06-19 02:24:37 on dist-test-slave-xkrt"
server_key: "fae3ed0ff4b10e79d225662357f264e1"
server_key_iv: "c43a7f43b3ab899738a46d42c338338e"
server_key_version: "encryptionkey@0"
I20260619 02:24:37.372921 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:24:37.375291 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:37.376456 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:24:37.376595 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "1033e8b8bd66449291282189e3a99d8d"
format_stamp: "Formatted at 2026-06-19 02:24:37 on dist-test-slave-xkrt"
server_key: "fae3ed0ff4b10e79d225662357f264e1"
server_key_iv: "c43a7f43b3ab899738a46d42c338338e"
server_key_version: "encryptionkey@0"
I20260619 02:24:37.376700 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:37.390820 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:37.393810 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:37.394001 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:37.400766 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44819
I20260619 02:24:37.400831 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44819 every 8 connection(s)
I20260619 02:24:37.401703 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:24:37.404306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 930
I20260619 02:24:37.404423 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:24:37.404423 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:37.404691 392 external_mini_cluster.cc:1428] Setting key d0c9c725de9b2453f80f4c097dd84ecb
I20260619 02:24:37.411109 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d: Bootstrap starting.
Jun 19 02:24:37 dist-test-slave-xkrt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:37.413755 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:37.414451 998 log.cc:824] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:37.416198 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d: No bootstrap required, opened a new log
I20260619 02:24:37.418388 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:37.405973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.0.98.62:44819)
0619 02:24:37.406351 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:37.406366 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:37.406399 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:24:37.407084 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:37.407101 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:37.407405 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:37.407665 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:37.408566 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:37.409513 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:37.410699 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:37.411138 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:37.411334 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:37.414048 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:37.414052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:37.414065 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:37.414090 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:37.415758 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:37.416199 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:37.416209 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:37.416215 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:37.416350 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:37.416636 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:37.416639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:37.416640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:37.416992 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:37.417181 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:37.417454 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 02:24:37.417748 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":121,"threads_started":1}
I20260619 02:24:37.419831 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1033e8b8bd66449291282189e3a99d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44819 } }
I20260619 02:24:37.420078 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:37.420133 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1033e8b8bd66449291282189e3a99d8d, State: Initialized, Role: FOLLOWER
I20260619 02:24:37.420598 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1033e8b8bd66449291282189e3a99d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44819 } }
I20260619 02:24:37.420773 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:24:37.420828 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:24:37.420900 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:37.421705 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1033e8b8bd66449291282189e3a99d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44819 } }
I20260619 02:24:37.422045 998 leader_election.cc:302] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1033e8b8bd66449291282189e3a99d8d; no voters:
I20260619 02:24:37.422354 998 leader_election.cc:288] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:24:37.422461 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:37.422665 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [term 1 LEADER]: Becoming Leader. State: Replica: 1033e8b8bd66449291282189e3a99d8d, State: Running, Role: LEADER
I20260619 02:24:37.422941 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1033e8b8bd66449291282189e3a99d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44819 } }
I20260619 02:24:37.423539 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:24:37.424497 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1033e8b8bd66449291282189e3a99d8d. Latest consensus state: current_term: 1 leader_uuid: "1033e8b8bd66449291282189e3a99d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1033e8b8bd66449291282189e3a99d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44819 } } }
I20260619 02:24:37.424618 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:37.424896 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1033e8b8bd66449291282189e3a99d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1033e8b8bd66449291282189e3a99d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44819 } } }
I20260619 02:24:37.424985 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:37.425130 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:24:37.428344 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:24:37.434338 1012 catalog_manager.cc:1347] Generated new cluster ID: 3df6e751ceca490e86d95e22fb4c2623
I20260619 02:24:37.434422 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:24:37.440016 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:24:37.440902 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:24:37.445097 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1033e8b8bd66449291282189e3a99d8d: Generated new TSK 0
I20260619 02:24:37.445787 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260619 02:24:38.517870 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 1.067s user 0.000s sys 0.003s
I20260619 02:24:38.518366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44819
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:38.633075 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:38.633311 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:38.633394 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:38.637319 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:38.637370 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:38.637468 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:24:38.642612 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44819
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:38.643616 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:38.644608 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:38.654042 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:38.654000 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:38.654052 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:38.654258 1024 server_base.cc:1034] running on GCE node
I20260619 02:24:38.654699 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:38.655336 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:38.656503 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781835878656485 us; error 28 us; skew 500 ppm
Jun 19 02:24:38 dist-test-slave-xkrt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:38.659195 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:24:38.660053 1024 webserver.cc:460] Webserver started at http://127.0.98.1:39547/ using document root <none> and password file <none>
I20260619 02:24:38.660607 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:38.660665 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:38.660872 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:38.662914 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "22e09c4924514c08a8c2662437759882"
format_stamp: "Formatted at 2026-06-19 02:24:38 on dist-test-slave-xkrt"
server_key: "2e381b2191565ff7e9f9c93733a965ef"
server_key_iv: "34b4e066951d63d318f00938e72dcd5c"
server_key_version: "encryptionkey@0"
I20260619 02:24:38.663481 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "22e09c4924514c08a8c2662437759882"
format_stamp: "Formatted at 2026-06-19 02:24:38 on dist-test-slave-xkrt"
server_key: "2e381b2191565ff7e9f9c93733a965ef"
server_key_iv: "34b4e066951d63d318f00938e72dcd5c"
server_key_version: "encryptionkey@0"
I20260619 02:24:38.667079 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:24:38.669440 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:38.670694 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:24:38.670852 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "22e09c4924514c08a8c2662437759882"
format_stamp: "Formatted at 2026-06-19 02:24:38 on dist-test-slave-xkrt"
server_key: "2e381b2191565ff7e9f9c93733a965ef"
server_key_iv: "34b4e066951d63d318f00938e72dcd5c"
server_key_version: "encryptionkey@0"
I20260619 02:24:38.670962 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:38.691167 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:38.693884 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:38.694075 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:38.694621 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:38.695401 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:38.695472 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:38.695541 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:38.695595 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:38.704034 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35103
I20260619 02:24:38.704120 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35103 every 8 connection(s)
I20260619 02:24:38.704914 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:24:38 dist-test-slave-xkrt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:38.714059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 1024
I20260619 02:24:38.714175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:24:38.714414 392 external_mini_cluster.cc:1428] Setting key 0412310bbb7c75ddc3d3e31d19834fc5
I20260619 02:24:38.718626 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:38.707426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50721 (local address 127.0.98.62:44819)
0619 02:24:38.707685 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:38.707692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:38.707711 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:24:38.708976 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:38.708979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:38.709024 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:38.709105 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:38.710229 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:38.710774 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:38.711881 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:38.712090 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:38.712162 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:38.714702 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:38.714706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:38.714709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:38.714735 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:38.716522 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:38.717004 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:38.717010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:38.717015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:38.717112 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:38.717435 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:38.717440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:38.717442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:38.717698 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:38.717800 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:38.718330 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 02:24:38.718452 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260619 02:24:38.719414 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:38.706956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44819 (local address 127.0.98.1:50721)
0619 02:24:38.707462 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:38.707521 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:38.708793 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:38.709256 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:38.709263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:38.709622 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:38.710064 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:38.710076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:38.710881 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:38.710884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:38.711780 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:38.711789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:38.712210 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:38.712213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:38.712368 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:38.712913 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:38.712928 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:38.714537 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:38.716650 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:38.716654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:38.716665 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:38.716909 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:38.717208 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:38.717210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:38.717211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:38.717331 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:38.717809 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:38.717818 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:38.718043 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 02:24:38.718291 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 02:24:38.718601 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260619 02:24:38.720525 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44819
I20260619 02:24:38.720784 1153 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:38.721242 1153 heartbeater.cc:511] Master 127.0.98.62:44819 requested a full tablet report, sending...
I20260619 02:24:38.722750 962 ts_manager.cc:194] Registered new tserver with Master: 22e09c4924514c08a8c2662437759882 (127.0.98.1:35103)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:24:38.723969 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260619 02:24:39.676087 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.960s user 0.000s sys 0.003s
I20260619 02:24:39.676620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44819
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:24:39.726325 1153 heartbeater.cc:503] Master 127.0.98.62:44819 was elected leader, sending a full tablet report...
W20260619 02:24:39.789320 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:39.789630 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:39.789686 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:39.793740 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:39.793790 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:39.793891 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:24:39.799018 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44819
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:39.800072 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:39.801049 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:39.810535 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:39.810663 1159 server_base.cc:1034] running on GCE node
W20260619 02:24:39.810539 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:39.810604 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:39.811173 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:39.811841 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:39.813030 1159 hybrid_clock.cc:648] HybridClock initialized: now 1781835879813011 us; error 30 us; skew 500 ppm
Jun 19 02:24:39 dist-test-slave-xkrt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:39.815738 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:24:39.816583 1159 webserver.cc:460] Webserver started at http://127.0.98.2:36319/ using document root <none> and password file <none>
I20260619 02:24:39.817116 1159 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:39.817209 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:39.817425 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:39.819396 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "31be07f3440f4a188cf541f19b869737"
format_stamp: "Formatted at 2026-06-19 02:24:39 on dist-test-slave-xkrt"
server_key: "892692f18918b7f73c4d353a1c7dfbc7"
server_key_iv: "3992775beed79de9ffe0b15af2da3fe0"
server_key_version: "encryptionkey@0"
I20260619 02:24:39.819938 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "31be07f3440f4a188cf541f19b869737"
format_stamp: "Formatted at 2026-06-19 02:24:39 on dist-test-slave-xkrt"
server_key: "892692f18918b7f73c4d353a1c7dfbc7"
server_key_iv: "3992775beed79de9ffe0b15af2da3fe0"
server_key_version: "encryptionkey@0"
I20260619 02:24:39.823570 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:24:39.826045 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:39.827143 1159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:24:39.827272 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "31be07f3440f4a188cf541f19b869737"
format_stamp: "Formatted at 2026-06-19 02:24:39 on dist-test-slave-xkrt"
server_key: "892692f18918b7f73c4d353a1c7dfbc7"
server_key_iv: "3992775beed79de9ffe0b15af2da3fe0"
server_key_version: "encryptionkey@0"
I20260619 02:24:39.827375 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:39.843242 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:39.845952 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:39.846169 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:39.846717 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:39.847474 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:39.847543 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:39.847613 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:39.847657 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:39.856928 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44643
I20260619 02:24:39.856961 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44643 every 8 connection(s)
I20260619 02:24:39.857839 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:24:39.862630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 1159
I20260619 02:24:39.862785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:24:39.863062 392 external_mini_cluster.cc:1428] Setting key a30cb8dba3329ddd16671f103657d1ed
Jun 19 02:24:39 dist-test-slave-xkrt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:24:39.872988 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:39.859545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47601 (local address 127.0.98.62:44819)
0619 02:24:39.859785 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:39.859788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:39.860489 (+ 701us) server_negotiation.cc:407] Connection header received
0619 02:24:39.861858 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:39.861862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:39.861922 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:39.862034 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:39.863266 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:39.863964 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:39.865645 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:39.865875 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:39.865986 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:39.868528 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:39.868534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:39.868539 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:39.868571 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:39.870659 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:39.871330 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:39.871334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:39.871336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:39.871430 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:39.871798 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:39.871803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:39.871806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:39.872063 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:39.872176 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:39.872738 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 02:24:39.872856 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
I20260619 02:24:39.873848 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:39.859887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44819 (local address 127.0.98.2:47601)
0619 02:24:39.860350 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:39.860391 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:39.861658 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:39.862201 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:39.862208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:39.862557 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:39.863088 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:39.863103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:39.864082 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:39.864085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:39.865486 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:39.865500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:39.866036 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:39.866042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:39.866248 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:39.866832 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:39.866848 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:39.868307 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:39.870855 (+ 2548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:39.870860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:39.870888 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:39.871222 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:39.871539 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:39.871543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:39.871545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:39.871702 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:39.872171 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:39.872176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:39.872438 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 02:24:39.872675 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 02:24:39.872931 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":114,"threads_started":1}
I20260619 02:24:39.875046 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44819
I20260619 02:24:39.875312 1288 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:39.875784 1288 heartbeater.cc:511] Master 127.0.98.62:44819 requested a full tablet report, sending...
I20260619 02:24:39.876754 962 ts_manager.cc:194] Registered new tserver with Master: 31be07f3440f4a188cf541f19b869737 (127.0.98.2:44643)
I20260619 02:24:39.877326 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260619 02:24:40.573948 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.709s user 0.000s sys 0.004s
I20260619 02:24:40.574483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44819
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:40.684736 1294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:40.684986 1294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:40.685029 1294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:40.688987 1294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:40.689042 1294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:40.689101 1294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:24:40.694180 1294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44819
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:40.695106 1294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:40.696029 1294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:40.705268 1300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:40.705430 1294 server_base.cc:1034] running on GCE node
W20260619 02:24:40.705279 1302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:40.705772 1299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:40.706192 1294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:40.706866 1294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:40.708015 1294 hybrid_clock.cc:648] HybridClock initialized: now 1781835880707985 us; error 42 us; skew 500 ppm
Jun 19 02:24:40 dist-test-slave-xkrt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:40.710768 1294 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:24:40.711633 1294 webserver.cc:460] Webserver started at http://127.0.98.3:41101/ using document root <none> and password file <none>
I20260619 02:24:40.712168 1294 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:40.712263 1294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:40.712479 1294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:40.714479 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "0700abedb8a440e88d7f2713d2254a79"
format_stamp: "Formatted at 2026-06-19 02:24:40 on dist-test-slave-xkrt"
server_key: "cbf8aa450a5a16a93f24b7ada7c7a84a"
server_key_iv: "06d83bafab30bb615bd92d73b88ffa05"
server_key_version: "encryptionkey@0"
I20260619 02:24:40.715019 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0700abedb8a440e88d7f2713d2254a79"
format_stamp: "Formatted at 2026-06-19 02:24:40 on dist-test-slave-xkrt"
server_key: "cbf8aa450a5a16a93f24b7ada7c7a84a"
server_key_iv: "06d83bafab30bb615bd92d73b88ffa05"
server_key_version: "encryptionkey@0"
I20260619 02:24:40.718494 1294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:24:40.720736 1309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:40.722056 1294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:24:40.722184 1294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "0700abedb8a440e88d7f2713d2254a79"
format_stamp: "Formatted at 2026-06-19 02:24:40 on dist-test-slave-xkrt"
server_key: "cbf8aa450a5a16a93f24b7ada7c7a84a"
server_key_iv: "06d83bafab30bb615bd92d73b88ffa05"
server_key_version: "encryptionkey@0"
I20260619 02:24:40.722289 1294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:40.736320 1294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:40.739612 1294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:40.739810 1294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:40.740397 1294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:40.741227 1294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:40.741302 1294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:40.741372 1294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:40.741420 1294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:40.749995 1294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41817
I20260619 02:24:40.750043 1422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41817 every 8 connection(s)
I20260619 02:24:40.751066 1294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:24:40 dist-test-slave-xkrt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:40.760828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 1294
I20260619 02:24:40.760936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:24:40.761211 392 external_mini_cluster.cc:1428] Setting key e1d2806f20703c83150e9d878ded8260
I20260619 02:24:40.765321 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.752908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59461 (local address 127.0.98.62:44819)
0619 02:24:40.753189 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:40.753192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:40.753936 (+ 744us) server_negotiation.cc:407] Connection header received
0619 02:24:40.755393 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:40.755396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:40.755446 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:40.755531 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:40.756943 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.757493 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.758669 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.758916 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.759002 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.761474 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:40.761478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:40.761481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:40.761510 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:40.763464 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:40.763959 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:40.763962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:40.763963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:40.764041 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:40.764404 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:40.764406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:40.764407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:40.764570 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:40.764645 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:40.765099 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 02:24:40.765195 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":105,"threads_started":1}
I20260619 02:24:40.766182 1426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.753317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44819 (local address 127.0.98.3:59461)
0619 02:24:40.753790 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:40.753831 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:40.755047 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:40.755968 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:40.755975 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:40.756308 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:40.756800 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:40.756813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.757667 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.757669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:40.758543 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:40.758552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.759044 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.759047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:40.759177 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.759706 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:40.759721 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:40.761315 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:40.763596 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:40.763600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:40.763611 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:40.763873 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:40.764155 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:40.764158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:40.764160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:40.764313 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:40.764655 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:40.764660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:40.764901 (+ 241us) client_negotiation.cc:769] Sending connection context
0619 02:24:40.765057 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 02:24:40.765305 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":112,"threads_started":1}
I20260619 02:24:40.767370 1423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44819
I20260619 02:24:40.767598 1423 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:40.768072 1423 heartbeater.cc:511] Master 127.0.98.62:44819 requested a full tablet report, sending...
I20260619 02:24:40.769048 962 ts_manager.cc:194] Registered new tserver with Master: 0700abedb8a440e88d7f2713d2254a79 (127.0.98.3:41817)
I20260619 02:24:40.769657 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59461
I20260619 02:24:40.775416 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:24:40.786186 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.778920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.0.98.62:44819)
0619 02:24:40.779050 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:40.779054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:40.779236 (+ 182us) server_negotiation.cc:407] Connection header received
0619 02:24:40.779370 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:40.779372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:40.779411 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:40.779481 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:40.780163 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.780773 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.781517 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.781842 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.781936 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.783134 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:40.783137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:40.783139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:40.783171 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:40.784642 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:40.784987 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:40.784989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:40.784991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:40.785059 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:40.785272 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:40.785274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:40.785295 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:40.785503 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:40.785685 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:40.785881 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 02:24:40.786010 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:24:40.791085 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:24:40.794219 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:24:40.809094 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.803594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.0.98.2:44643)
0619 02:24:40.804268 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:40.804275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:40.804294 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:24:40.804365 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:40.804370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:40.804576 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:40.805071 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:40.805207 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.805986 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.807759 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.808418 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.808666 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.808807 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:40.808856 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 02:24:40.808949 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":64,"threads_started":1}
I20260619 02:24:40.809114 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.803218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44643 (local address 127.0.0.1:57516)
0619 02:24:40.803694 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:40.803728 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:40.803865 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:40.804723 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:40.804728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:40.804749 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:40.804933 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:40.804941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.806096 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.806102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:40.807627 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:40.807634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.808560 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.808563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:40.808690 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.808728 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 02:24:40.808829 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 02:24:40.808941 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":249,"spinlock_wait_cycles":5248,"thread_start_us":82,"threads_started":1}
I20260619 02:24:40.809142 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.802837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41817 (local address 127.0.0.1:37856)
0619 02:24:40.803680 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:40.803729 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:40.803865 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:40.804604 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:40.804609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:40.804629 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:40.804764 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:40.804774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.806224 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.806227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:40.807615 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:40.807626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.808733 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.808737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:40.808810 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.808826 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 02:24:40.808916 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 02:24:40.808982 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":144,"thread_start_us":102,"threads_started":1}
I20260619 02:24:40.809167 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.802965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.0.98.3:41817)
0619 02:24:40.803565 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:40.803576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:40.803808 (+ 232us) server_negotiation.cc:407] Connection header received
0619 02:24:40.804035 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:40.804039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:40.804260 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:40.804398 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:40.804887 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.806104 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.807741 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.808493 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.808577 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.808676 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:40.808945 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 02:24:40.809033 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":127,"threads_started":1}
I20260619 02:24:40.809787 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.802953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35103 (local address 127.0.0.1:46620)
0619 02:24:40.803689 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:40.803731 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:40.803865 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:40.804783 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:40.804786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:40.804798 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:40.804969 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:40.804978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.806329 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.806333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:40.807458 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:40.807467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.808529 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.808536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:40.809471 (+ 935us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.809493 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:24:40.809586 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 02:24:40.809649 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":80,"spinlock_wait_cycles":5504,"thread_start_us":68,"threads_started":1}
I20260619 02:24:40.809856 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.803948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.0.98.1:35103)
0619 02:24:40.804390 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:40.804396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:40.804416 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:24:40.804484 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:40.804489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:40.804664 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:40.804816 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:40.805084 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.806166 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.807573 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.808398 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.808511 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.808662 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:40.809634 (+ 972us) server_negotiation.cc:299] Negotiation successful
0619 02:24:40.809726 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":129,"threads_started":1}
I20260619 02:24:40.812177 1222 tablet_service.cc:1467] Processing CreateTablet for tablet a456897bc6154061b2ca5a6073617617 (DEFAULT_TABLE table=test-table [id=d3ab9d35bfd545a988dcf06880d37fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:40.812197 1357 tablet_service.cc:1467] Processing CreateTablet for tablet a456897bc6154061b2ca5a6073617617 (DEFAULT_TABLE table=test-table [id=d3ab9d35bfd545a988dcf06880d37fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:40.812723 1087 tablet_service.cc:1467] Processing CreateTablet for tablet a456897bc6154061b2ca5a6073617617 (DEFAULT_TABLE table=test-table [id=d3ab9d35bfd545a988dcf06880d37fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:40.813118 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a456897bc6154061b2ca5a6073617617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:40.813160 1357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a456897bc6154061b2ca5a6073617617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:40.813575 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a456897bc6154061b2ca5a6073617617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:40.819530 1442 tablet_bootstrap.cc:492] T a456897bc6154061b2ca5a6073617617 P 31be07f3440f4a188cf541f19b869737: Bootstrap starting.
I20260619 02:24:40.820582 1443 tablet_bootstrap.cc:492] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882: Bootstrap starting.
I20260619 02:24:40.822657 1443 tablet_bootstrap.cc:654] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:40.822657 1442 tablet_bootstrap.cc:654] T a456897bc6154061b2ca5a6073617617 P 31be07f3440f4a188cf541f19b869737: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:40.823956 1444 tablet_bootstrap.cc:492] T a456897bc6154061b2ca5a6073617617 P 0700abedb8a440e88d7f2713d2254a79: Bootstrap starting.
I20260619 02:24:40.823982 1443 log.cc:824] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:40.824009 1442 log.cc:824] T a456897bc6154061b2ca5a6073617617 P 31be07f3440f4a188cf541f19b869737: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:40.826050 1443 tablet_bootstrap.cc:492] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882: No bootstrap required, opened a new log
I20260619 02:24:40.826062 1444 tablet_bootstrap.cc:654] T a456897bc6154061b2ca5a6073617617 P 0700abedb8a440e88d7f2713d2254a79: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:40.826262 1443 ts_tablet_manager.cc:1397] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:24:40.826640 1442 tablet_bootstrap.cc:492] T a456897bc6154061b2ca5a6073617617 P 31be07f3440f4a188cf541f19b869737: No bootstrap required, opened a new log
I20260619 02:24:40.826843 1442 ts_tablet_manager.cc:1397] T a456897bc6154061b2ca5a6073617617 P 31be07f3440f4a188cf541f19b869737: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 02:24:40.827395 1444 log.cc:824] T a456897bc6154061b2ca5a6073617617 P 0700abedb8a440e88d7f2713d2254a79: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:40.829377 1444 tablet_bootstrap.cc:492] T a456897bc6154061b2ca5a6073617617 P 0700abedb8a440e88d7f2713d2254a79: No bootstrap required, opened a new log
I20260619 02:24:40.829533 1444 ts_tablet_manager.cc:1397] T a456897bc6154061b2ca5a6073617617 P 0700abedb8a440e88d7f2713d2254a79: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:24:40.830080 1443 raft_consensus.cc:348] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700abedb8a440e88d7f2713d2254a79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41817 } } peers { permanent_uuid: "22e09c4924514c08a8c2662437759882" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "31be07f3440f4a188cf541f19b869737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44643 } }
I20260619 02:24:40.830320 1443 raft_consensus.cc:374] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:40.830408 1443 raft_consensus.cc:729] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22e09c4924514c08a8c2662437759882, State: Initialized, Role: FOLLOWER
I20260619 02:24:40.830901 1443 consensus_queue.cc:260] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700abedb8a440e88d7f2713d2254a79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41817 } } peers { permanent_uuid: "22e09c4924514c08a8c2662437759882" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "31be07f3440f4a188cf541f19b869737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44643 } }
I20260619 02:24:40.830953 1442 raft_consensus.cc:348] T a456897bc6154061b2ca5a6073617617 P 31be07f3440f4a188cf541f19b869737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700abedb8a440e88d7f2713d2254a79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41817 } } peers { permanent_uuid: "22e09c4924514c08a8c2662437759882" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "31be07f3440f4a188cf541f19b869737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44643 } }
I20260619 02:24:40.831235 1442 raft_consensus.cc:374] T a456897bc6154061b2ca5a6073617617 P 31be07f3440f4a188cf541f19b869737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:40.831319 1442 raft_consensus.cc:729] T a456897bc6154061b2ca5a6073617617 P 31be07f3440f4a188cf541f19b869737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31be07f3440f4a188cf541f19b869737, State: Initialized, Role: FOLLOWER
I20260619 02:24:40.831754 1442 consensus_queue.cc:260] T a456897bc6154061b2ca5a6073617617 P 31be07f3440f4a188cf541f19b869737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700abedb8a440e88d7f2713d2254a79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41817 } } peers { permanent_uuid: "22e09c4924514c08a8c2662437759882" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "31be07f3440f4a188cf541f19b869737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44643 } }
I20260619 02:24:40.832154 1443 ts_tablet_manager.cc:1428] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:24:40.832417 1288 heartbeater.cc:503] Master 127.0.98.62:44819 was elected leader, sending a full tablet report...
I20260619 02:24:40.832683 1442 ts_tablet_manager.cc:1428] T a456897bc6154061b2ca5a6073617617 P 31be07f3440f4a188cf541f19b869737: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:24:40.833292 1444 raft_consensus.cc:348] T a456897bc6154061b2ca5a6073617617 P 0700abedb8a440e88d7f2713d2254a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700abedb8a440e88d7f2713d2254a79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41817 } } peers { permanent_uuid: "22e09c4924514c08a8c2662437759882" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "31be07f3440f4a188cf541f19b869737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44643 } }
I20260619 02:24:40.833629 1444 raft_consensus.cc:374] T a456897bc6154061b2ca5a6073617617 P 0700abedb8a440e88d7f2713d2254a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:40.833695 1444 raft_consensus.cc:729] T a456897bc6154061b2ca5a6073617617 P 0700abedb8a440e88d7f2713d2254a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0700abedb8a440e88d7f2713d2254a79, State: Initialized, Role: FOLLOWER
I20260619 02:24:40.834059 1444 consensus_queue.cc:260] T a456897bc6154061b2ca5a6073617617 P 0700abedb8a440e88d7f2713d2254a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700abedb8a440e88d7f2713d2254a79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41817 } } peers { permanent_uuid: "22e09c4924514c08a8c2662437759882" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "31be07f3440f4a188cf541f19b869737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44643 } }
I20260619 02:24:40.834671 1423 heartbeater.cc:503] Master 127.0.98.62:44819 was elected leader, sending a full tablet report...
I20260619 02:24:40.834991 1444 ts_tablet_manager.cc:1428] T a456897bc6154061b2ca5a6073617617 P 0700abedb8a440e88d7f2713d2254a79: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 02:24:40.859270 1289 tablet.cc:2369] T a456897bc6154061b2ca5a6073617617 P 31be07f3440f4a188cf541f19b869737: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:24:40.872257 1448 raft_consensus.cc:482] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:24:40.872468 1448 raft_consensus.cc:504] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700abedb8a440e88d7f2713d2254a79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41817 } } peers { permanent_uuid: "22e09c4924514c08a8c2662437759882" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "31be07f3440f4a188cf541f19b869737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44643 } }
I20260619 02:24:40.873775 1448 leader_election.cc:288] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0700abedb8a440e88d7f2713d2254a79 (127.0.98.3:41817), 31be07f3440f4a188cf541f19b869737 (127.0.98.2:44643)
I20260619 02:24:40.878149 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.874104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33073 (local address 127.0.98.2:44643)
0619 02:24:40.874255 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:40.874261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:40.874584 (+ 323us) server_negotiation.cc:407] Connection header received
0619 02:24:40.874735 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:40.874744 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:40.874785 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:40.874862 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:40.875172 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.875803 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.877145 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.877599 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.877720 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.877784 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:40.877964 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 02:24:40.878031 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:24:40.878180 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.874042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44643 (local address 127.0.98.1:33073)
0619 02:24:40.874486 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:40.874497 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:40.874621 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:40.874883 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:40.874886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:40.874911 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:40.875066 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:40.875071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.875904 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.875908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:40.876978 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:40.876988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.877724 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.877727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:40.877823 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.877845 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:24:40.877951 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 02:24:40.878020 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":133,"spinlock_wait_cycles":25856,"thread_start_us":144,"threads_started":1}
I20260619 02:24:40.878901 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.874100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51137 (local address 127.0.98.3:41817)
0619 02:24:40.874257 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:40.874264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:40.874547 (+ 283us) server_negotiation.cc:407] Connection header received
0619 02:24:40.874725 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:40.874729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:40.874783 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:40.874875 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:40.875194 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.875814 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.877400 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.877947 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.878076 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.878138 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:40.878704 (+ 566us) server_negotiation.cc:299] Negotiation successful
0619 02:24:40.878789 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:24:40.878916 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.874036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41817 (local address 127.0.98.1:51137)
0619 02:24:40.874441 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:40.874455 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:40.874603 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:40.874885 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:40.874888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:40.874919 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:40.875068 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:40.875073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.875968 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.875974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:40.877268 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:40.877278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.878076 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.878080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:40.878185 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.878212 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 02:24:40.878676 (+ 464us) client_negotiation.cc:240] Negotiation successful
0619 02:24:40.878749 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":18,"spinlock_wait_cycles":7168,"thread_start_us":200,"threads_started":1}
I20260619 02:24:40.879050 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a456897bc6154061b2ca5a6073617617" candidate_uuid: "22e09c4924514c08a8c2662437759882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31be07f3440f4a188cf541f19b869737" is_pre_election: true
I20260619 02:24:40.879379 1242 raft_consensus.cc:2454] T a456897bc6154061b2ca5a6073617617 P 31be07f3440f4a188cf541f19b869737 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22e09c4924514c08a8c2662437759882 in term 0.
I20260619 02:24:40.879549 1377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a456897bc6154061b2ca5a6073617617" candidate_uuid: "22e09c4924514c08a8c2662437759882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0700abedb8a440e88d7f2713d2254a79" is_pre_election: true
I20260619 02:24:40.879861 1377 raft_consensus.cc:2454] T a456897bc6154061b2ca5a6073617617 P 0700abedb8a440e88d7f2713d2254a79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22e09c4924514c08a8c2662437759882 in term 0.
I20260619 02:24:40.879894 1041 leader_election.cc:302] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22e09c4924514c08a8c2662437759882, 31be07f3440f4a188cf541f19b869737; no voters:
I20260619 02:24:40.880223 1448 raft_consensus.cc:2781] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:24:40.880335 1448 raft_consensus.cc:482] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:24:40.880411 1448 raft_consensus.cc:3037] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:40.896934 1448 raft_consensus.cc:504] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700abedb8a440e88d7f2713d2254a79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41817 } } peers { permanent_uuid: "22e09c4924514c08a8c2662437759882" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "31be07f3440f4a188cf541f19b869737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44643 } }
I20260619 02:24:40.897423 1448 leader_election.cc:288] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [CANDIDATE]: Term 1 election: Requested vote from peers 0700abedb8a440e88d7f2713d2254a79 (127.0.98.3:41817), 31be07f3440f4a188cf541f19b869737 (127.0.98.2:44643)
I20260619 02:24:40.897835 1377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a456897bc6154061b2ca5a6073617617" candidate_uuid: "22e09c4924514c08a8c2662437759882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0700abedb8a440e88d7f2713d2254a79"
I20260619 02:24:40.897835 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a456897bc6154061b2ca5a6073617617" candidate_uuid: "22e09c4924514c08a8c2662437759882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31be07f3440f4a188cf541f19b869737"
I20260619 02:24:40.897977 1242 raft_consensus.cc:3037] T a456897bc6154061b2ca5a6073617617 P 31be07f3440f4a188cf541f19b869737 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:40.897977 1377 raft_consensus.cc:3037] T a456897bc6154061b2ca5a6073617617 P 0700abedb8a440e88d7f2713d2254a79 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:40.899262 1377 raft_consensus.cc:2454] T a456897bc6154061b2ca5a6073617617 P 0700abedb8a440e88d7f2713d2254a79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22e09c4924514c08a8c2662437759882 in term 1.
I20260619 02:24:40.899262 1242 raft_consensus.cc:2454] T a456897bc6154061b2ca5a6073617617 P 31be07f3440f4a188cf541f19b869737 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22e09c4924514c08a8c2662437759882 in term 1.
I20260619 02:24:40.899609 1040 leader_election.cc:302] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0700abedb8a440e88d7f2713d2254a79, 22e09c4924514c08a8c2662437759882; no voters:
I20260619 02:24:40.899797 1448 raft_consensus.cc:2781] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:40.900063 1448 raft_consensus.cc:686] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [term 1 LEADER]: Becoming Leader. State: Replica: 22e09c4924514c08a8c2662437759882, State: Running, Role: LEADER
I20260619 02:24:40.900472 1448 consensus_queue.cc:237] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700abedb8a440e88d7f2713d2254a79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41817 } } peers { permanent_uuid: "22e09c4924514c08a8c2662437759882" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "31be07f3440f4a188cf541f19b869737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44643 } }
I20260619 02:24:40.903936 962 catalog_manager.cc:5495] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22e09c4924514c08a8c2662437759882 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "22e09c4924514c08a8c2662437759882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700abedb8a440e88d7f2713d2254a79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22e09c4924514c08a8c2662437759882" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31be07f3440f4a188cf541f19b869737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44643 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:24:40 dist-test-slave-xkrt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 02:24:40.933405 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.924088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.0.98.1:35103)
0619 02:24:40.924248 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:40.924255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:40.924372 (+ 117us) server_negotiation.cc:407] Connection header received
0619 02:24:40.924534 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:40.924537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:40.924579 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:40.924648 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:40.925380 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.926025 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.926898 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.927104 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.927184 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.929640 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:40.929651 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:40.929667 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:40.929689 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:40.931375 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:40.931792 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:40.931801 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:40.931805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:40.931902 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:40.932183 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:40.932185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:40.932186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:40.932754 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:40.932923 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:40.933169 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 02:24:40.933265 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 02:24:40 dist-test-slave-xkrt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835880, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260619 02:24:40.957427 1154 tablet.cc:2369] T a456897bc6154061b2ca5a6073617617 P 22e09c4924514c08a8c2662437759882: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 02:24:40 dist-test-slave-xkrt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835880, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 02:24:40.996662 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:40.987145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46642 (local address 127.0.98.1:35103)
0619 02:24:40.987310 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:40.987316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:40.987528 (+ 212us) server_negotiation.cc:407] Connection header received
0619 02:24:40.987721 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:40.987728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:40.987782 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:40.987881 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:40.988700 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.989251 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.990636 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:40.990878 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:40.991005 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:40.993375 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:40.993379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:40.993381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:40.993408 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:40.994917 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:40.995364 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:40.995369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:40.995371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:40.995460 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:40.995758 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:40.995763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:40.995764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:40.995942 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:40.996079 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:40.996367 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 02:24:40.996487 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 02:24:41.001981 1424 tablet.cc:2369] T a456897bc6154061b2ca5a6073617617 P 0700abedb8a440e88d7f2713d2254a79: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 02:24:41 dist-test-slave-xkrt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:24:41 dist-test-slave-xkrt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 02:24:41.084539 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:41.073491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46654 (local address 127.0.98.1:35103)
0619 02:24:41.073652 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:41.073657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:41.073848 (+ 191us) server_negotiation.cc:407] Connection header received
0619 02:24:41.074047 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:41.074051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:41.074111 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:41.074199 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:41.075310 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:41.076087 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:41.077631 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:41.077858 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:41.077945 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:41.080881 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:41.080887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:41.080891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:41.080925 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:41.083096 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:41.083550 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:41.083553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:41.083555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:41.083635 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:41.083874 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:41.083877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:41.083878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:41.084046 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:41.084143 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:41.084310 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 02:24:41.084410 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 02:24:41.086030 1067 tablet_service.cc:2908] Rejecting scan request for tablet a456897bc6154061b2ca5a6073617617: Uninitialized: clean time has not yet been initialized
I20260619 02:24:41.088183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 1024
I20260619 02:24:41.094345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 1159
I20260619 02:24:41.099659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 1294
I20260619 02:24:41.104811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 930
2026-06-19T02:24:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5027 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:24:42 dist-test-slave-xkrt krb5kdc[1467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:24:42 dist-test-slave-xkrt krb5kdc[1467](info): set up 2 sockets
Jun 19 02:24:42 dist-test-slave-xkrt krb5kdc[1467](info): commencing operation
krb5kdc: starting...
W20260619 02:24:43.185277 392 mini_kdc.cc:121] Time spent starting KDC: real 2.058s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:24:43 dist-test-slave-xkrt krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:24:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:24:43.304612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44977
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:43.413218 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:43.413463 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:43.413502 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:43.417429 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:24:43.417479 1480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:24:43.417496 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:43.417514 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:43.417531 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:24:43.417587 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:24:43.422832 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44977
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:43.423736 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:43.424642 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:43.432646 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:43.432627 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:43.432672 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:43.432899 1480 server_base.cc:1034] running on GCE node
I20260619 02:24:43.433591 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:43.434644 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:43.435830 1480 hybrid_clock.cc:648] HybridClock initialized: now 1781835883435816 us; error 35 us; skew 500 ppm
Jun 19 02:24:43 dist-test-slave-xkrt krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:43.442018 1480 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:24:43.442831 1480 webserver.cc:460] Webserver started at http://127.0.98.62:41565/ using document root <none> and password file <none>
I20260619 02:24:43.443336 1480 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:43.443400 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:43.443557 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:43.445466 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "cd2b3115c9084bf0807e632b291296aa"
format_stamp: "Formatted at 2026-06-19 02:24:43 on dist-test-slave-xkrt"
server_key: "8a93121cb1dc04e9749d3790a836a181"
server_key_iv: "f988c27d64e29223ade3af910e0d53af"
server_key_version: "encryptionkey@0"
I20260619 02:24:43.446039 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd2b3115c9084bf0807e632b291296aa"
format_stamp: "Formatted at 2026-06-19 02:24:43 on dist-test-slave-xkrt"
server_key: "8a93121cb1dc04e9749d3790a836a181"
server_key_iv: "f988c27d64e29223ade3af910e0d53af"
server_key_version: "encryptionkey@0"
I20260619 02:24:43.449481 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:24:43.451788 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:43.453003 1480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:24:43.453136 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "cd2b3115c9084bf0807e632b291296aa"
format_stamp: "Formatted at 2026-06-19 02:24:43 on dist-test-slave-xkrt"
server_key: "8a93121cb1dc04e9749d3790a836a181"
server_key_iv: "f988c27d64e29223ade3af910e0d53af"
server_key_version: "encryptionkey@0"
I20260619 02:24:43.453240 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:43.463932 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:43.466806 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:43.467002 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:43.474632 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44977
I20260619 02:24:43.474651 1557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44977 every 8 connection(s)
I20260619 02:24:43.475585 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:24:43.478536 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:43.480093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 1480
I20260619 02:24:43.480180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:24:43.480451 392 external_mini_cluster.cc:1428] Setting key a0b938369bf62ec35eb71dba821c8bab
I20260619 02:24:43.484803 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa: Bootstrap starting.
I20260619 02:24:43.487146 1558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:43.488059 1558 log.cc:824] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:24:43 dist-test-slave-xkrt krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:43.490515 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa: No bootstrap required, opened a new log
I20260619 02:24:43.494714 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:43.482013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.0.98.62:44977)
0619 02:24:43.482444 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:43.482458 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:43.482493 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:24:43.483096 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:43.483111 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:43.483430 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:43.483761 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:43.484527 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.485603 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.486457 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.486916 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.487083 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:43.490104 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:43.490109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:43.490121 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:43.490145 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:43.491889 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:43.492404 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:43.492408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:43.492412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:43.492521 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:43.492829 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:43.492832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:43.492833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:43.493181 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:43.493357 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:43.493684 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 02:24:43.493962 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":141,"threads_started":1}
I20260619 02:24:43.494750 1558 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2b3115c9084bf0807e632b291296aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44977 } }
I20260619 02:24:43.495078 1558 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:43.495157 1558 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2b3115c9084bf0807e632b291296aa, State: Initialized, Role: FOLLOWER
I20260619 02:24:43.495638 1558 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2b3115c9084bf0807e632b291296aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44977 } }
I20260619 02:24:43.495805 1558 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:24:43.495915 1558 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:24:43.496044 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:43.496989 1558 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2b3115c9084bf0807e632b291296aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44977 } }
I20260619 02:24:43.497330 1558 leader_election.cc:302] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd2b3115c9084bf0807e632b291296aa; no voters:
I20260619 02:24:43.497637 1558 leader_election.cc:288] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:24:43.497737 1563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:43.497969 1563 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [term 1 LEADER]: Becoming Leader. State: Replica: cd2b3115c9084bf0807e632b291296aa, State: Running, Role: LEADER
I20260619 02:24:43.498296 1563 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2b3115c9084bf0807e632b291296aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44977 } }
I20260619 02:24:43.498896 1558 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:24:43.499950 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd2b3115c9084bf0807e632b291296aa. Latest consensus state: current_term: 1 leader_uuid: "cd2b3115c9084bf0807e632b291296aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2b3115c9084bf0807e632b291296aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44977 } } }
I20260619 02:24:43.500097 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:43.500463 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd2b3115c9084bf0807e632b291296aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2b3115c9084bf0807e632b291296aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44977 } } }
I20260619 02:24:43.501212 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:24:43.501299 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:43.503772 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:24:43.509310 1572 catalog_manager.cc:1347] Generated new cluster ID: efdf6a5059a64616b6fef611478274d6
I20260619 02:24:43.509420 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:24:43.534291 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:24:43.535238 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:24:43.540620 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa: Generated new TSK 0
I20260619 02:24:43.541340 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:24:43.568696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44977
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:43.676667 1584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:43.676954 1584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:43.677027 1584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:43.680975 1584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:43.681068 1584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:43.681144 1584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:24:43.681233 1584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:24:43.686362 1584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:43.687363 1584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:43.688373 1584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:43.697757 1590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:43.697764 1592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:43.697824 1589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:43.697985 1584 server_base.cc:1034] running on GCE node
I20260619 02:24:43.698385 1584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:43.699012 1584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:43.700172 1584 hybrid_clock.cc:648] HybridClock initialized: now 1781835883700141 us; error 53 us; skew 500 ppm
Jun 19 02:24:43 dist-test-slave-xkrt krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:43.703017 1584 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:24:43.703847 1584 webserver.cc:460] Webserver started at http://127.0.98.1:45043/ using document root <none> and password file <none>
I20260619 02:24:43.704353 1584 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:43.704409 1584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:43.704563 1584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:43.706569 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "fc9be1d70d814ab4b8ea118e4f144dbd"
format_stamp: "Formatted at 2026-06-19 02:24:43 on dist-test-slave-xkrt"
server_key: "7a69331be1d3bf572a2508981a83b2bf"
server_key_iv: "1379bcf9035633791f9e0c416e5ebfe5"
server_key_version: "encryptionkey@0"
I20260619 02:24:43.707062 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fc9be1d70d814ab4b8ea118e4f144dbd"
format_stamp: "Formatted at 2026-06-19 02:24:43 on dist-test-slave-xkrt"
server_key: "7a69331be1d3bf572a2508981a83b2bf"
server_key_iv: "1379bcf9035633791f9e0c416e5ebfe5"
server_key_version: "encryptionkey@0"
I20260619 02:24:43.710618 1584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:24:43.713125 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:43.714437 1584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:24:43.714586 1584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "fc9be1d70d814ab4b8ea118e4f144dbd"
format_stamp: "Formatted at 2026-06-19 02:24:43 on dist-test-slave-xkrt"
server_key: "7a69331be1d3bf572a2508981a83b2bf"
server_key_iv: "1379bcf9035633791f9e0c416e5ebfe5"
server_key_version: "encryptionkey@0"
I20260619 02:24:43.714699 1584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:43.740097 1584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:43.743486 1584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:43.743688 1584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:43.745148 1584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:43.745273 1584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:43.745364 1584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:43.745414 1584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:24:43 dist-test-slave-xkrt krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:43.755921 1584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33835
I20260619 02:24:43.756313 1714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33835 every 8 connection(s)
I20260619 02:24:43.757050 1584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:24:43.759068 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:43.746025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45051 (local address 127.0.98.62:44977)
0619 02:24:43.746185 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:43.746192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:43.747052 (+ 860us) server_negotiation.cc:407] Connection header received
0619 02:24:43.748210 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:43.748215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:43.748284 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:43.748396 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:43.749890 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.750426 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.751631 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.751877 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.751999 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:43.754989 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:43.754995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:43.755002 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:43.755027 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:43.756829 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:43.757438 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:43.757441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:43.757443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:43.757578 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:43.757976 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:43.757981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:43.757983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:43.758171 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:43.758283 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:43.758760 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 02:24:43.758889 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:24:43.759831 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:43.746444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44977 (local address 127.0.98.1:45051)
0619 02:24:43.746881 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:43.746940 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:43.747926 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:43.748593 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:43.748601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:43.749097 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:43.749740 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:43.749756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.750551 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.750555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:43.751497 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:43.751507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.752047 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.752051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:43.752250 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:43.752831 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:43.752850 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:43.754809 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:43.756947 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:43.756951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:43.756961 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:43.757326 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:43.757689 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:43.757692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:43.757694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:43.757863 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:43.758280 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:43.758287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:43.758524 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 02:24:43.758705 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 02:24:43.759009 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":131,"threads_started":1}
I20260619 02:24:43.760991 1715 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44977
I20260619 02:24:43.761133 1715 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:43.761677 1715 heartbeater.cc:511] Master 127.0.98.62:44977 requested a full tablet report, sending...
I20260619 02:24:43.763108 1512 ts_manager.cc:194] Registered new tserver with Master: fc9be1d70d814ab4b8ea118e4f144dbd (127.0.98.1:33835)
I20260619 02:24:43.764549 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45051
I20260619 02:24:43.764789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 1584
I20260619 02:24:43.764910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:24:43.765223 392 external_mini_cluster.cc:1428] Setting key 50431931cbf9957d000f22b230a99895
I20260619 02:24:43.771047 1721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:43.763170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44977 (local address 127.0.98.1:41841)
0619 02:24:43.763556 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:43.763566 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:43.763646 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:43.763862 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:43.763864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:43.764030 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:43.764122 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:43.764127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.765069 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.765075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:43.765920 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:43.765932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.766462 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.766466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:43.766563 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:43.767092 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:43.767111 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:43.767593 (+ 482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:43.769582 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:43.769587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:43.769590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:43.769786 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:43.770084 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:43.770086 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:43.770088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:43.770162 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:43.770597 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:43.770599 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:43.770668 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 02:24:43.770749 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:24:43.770908 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":100,"threads_started":1}
I20260619 02:24:43.771173 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:43.763307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41841 (local address 127.0.98.62:44977)
0619 02:24:43.763440 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:43.763443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:43.763641 (+ 198us) server_negotiation.cc:407] Connection header received
0619 02:24:43.763724 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:43.763727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:43.763766 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:43.763840 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:43.764234 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.764917 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.766037 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.766333 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.766451 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:43.767727 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:43.767730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:43.767732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:43.767752 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:43.769426 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:43.769888 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:43.769894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:43.769896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:43.769986 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:43.770251 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:43.770257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:43.770261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:43.770505 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:43.770611 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:43.770837 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 02:24:43.770969 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:24:43.793357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44977
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:43.902266 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:43.902506 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:43.902549 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:43.906529 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:43.906582 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:43.906620 1724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:24:43.906651 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:24:43.911738 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:43.912659 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:43.913661 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:43.922698 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:43.922812 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:43.922857 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:43.923066 1724 server_base.cc:1034] running on GCE node
I20260619 02:24:43.923496 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:43.924106 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:43.925287 1724 hybrid_clock.cc:648] HybridClock initialized: now 1781835883925277 us; error 31 us; skew 500 ppm
Jun 19 02:24:43 dist-test-slave-xkrt krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:43.927980 1724 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:24:43.928850 1724 webserver.cc:460] Webserver started at http://127.0.98.2:41961/ using document root <none> and password file <none>
I20260619 02:24:43.929359 1724 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:43.929420 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:43.929636 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:43.931528 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "9319f9f48f8e435a9fba8793032f7966"
format_stamp: "Formatted at 2026-06-19 02:24:43 on dist-test-slave-xkrt"
server_key: "d73a8a66dfc708424b65b5c95a74dff4"
server_key_iv: "0ce91691a739a00470c50ab4f4c39ab7"
server_key_version: "encryptionkey@0"
I20260619 02:24:43.932027 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9319f9f48f8e435a9fba8793032f7966"
format_stamp: "Formatted at 2026-06-19 02:24:43 on dist-test-slave-xkrt"
server_key: "d73a8a66dfc708424b65b5c95a74dff4"
server_key_iv: "0ce91691a739a00470c50ab4f4c39ab7"
server_key_version: "encryptionkey@0"
I20260619 02:24:43.935519 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:24:43.937865 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:43.939088 1724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:24:43.939211 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "9319f9f48f8e435a9fba8793032f7966"
format_stamp: "Formatted at 2026-06-19 02:24:43 on dist-test-slave-xkrt"
server_key: "d73a8a66dfc708424b65b5c95a74dff4"
server_key_iv: "0ce91691a739a00470c50ab4f4c39ab7"
server_key_version: "encryptionkey@0"
I20260619 02:24:43.939311 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:43.969000 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:43.971869 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:43.972076 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:43.973470 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:43.973546 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:43.973663 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:43.973717 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:24:43 dist-test-slave-xkrt krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:43.984624 1724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45801
I20260619 02:24:43.985083 1854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45801 every 8 connection(s)
I20260619 02:24:43.985770 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:24:43.987804 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:43.974401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40341 (local address 127.0.98.62:44977)
0619 02:24:43.974522 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:43.974525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:43.975532 (+ 1007us) server_negotiation.cc:407] Connection header received
0619 02:24:43.976436 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:43.976440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:43.976495 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:43.976579 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:43.978163 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.978726 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.979943 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.980316 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.980435 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:43.983844 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:43.983848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:43.983850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:43.983881 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:43.985591 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:43.986115 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:43.986119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:43.986121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:43.986221 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:43.986755 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:43.986758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:43.986759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:43.986930 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:43.987043 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:43.987514 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 02:24:43.987634 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:24:43.988541 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:43.974933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44977 (local address 127.0.98.2:40341)
0619 02:24:43.975395 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:43.975434 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:43.976240 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:43.976864 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:43.976880 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:43.977388 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:43.978001 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:43.978041 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.978861 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.978864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:43.979806 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:43.979815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.980483 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.980487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:43.980680 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:43.981250 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:43.981275 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:43.983673 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:43.985694 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:43.985700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:43.985710 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:43.986012 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:43.986336 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:43.986340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:43.986342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:43.986518 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:43.987039 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:43.987045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:43.987282 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 02:24:43.987468 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 02:24:43.987769 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":141,"threads_started":1}
I20260619 02:24:43.989200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 1724
I20260619 02:24:43.989312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:24:43.989646 392 external_mini_cluster.cc:1428] Setting key fd10a04cf5ed2268614f9fe3705ef5de
I20260619 02:24:43.992050 1855 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44977
I20260619 02:24:43.992188 1855 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:43.992671 1855 heartbeater.cc:511] Master 127.0.98.62:44977 requested a full tablet report, sending...
I20260619 02:24:43.993680 1511 ts_manager.cc:194] Registered new tserver with Master: 9319f9f48f8e435a9fba8793032f7966 (127.0.98.2:45801)
I20260619 02:24:43.994463 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40341
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 02:24:44.003378 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:43.992265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44977 (local address 127.0.98.2:34751)
0619 02:24:43.993281 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:43.993295 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:43.993385 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:43.994884 (+ 1499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:43.994887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:43.996563 (+ 1676us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:43.996729 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:43.996742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.997482 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.997487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:43.998471 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:43.998485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.998939 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.998946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:43.999072 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:43.999613 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:43.999631 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:44.000107 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:44.001851 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:44.001853 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:44.001855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:44.002041 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:44.002375 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:44.002381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:44.002386 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:44.002486 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:44.002960 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:44.002962 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:44.003034 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.003115 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.003230 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":116,"threads_started":1}
I20260619 02:24:44.003579 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:43.992252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34751 (local address 127.0.98.62:44977)
0619 02:24:43.992417 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:43.992422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:43.993363 (+ 941us) server_negotiation.cc:407] Connection header received
0619 02:24:43.994748 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:43.994751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:43.994792 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:43.994865 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:43.996869 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.997347 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.998591 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:43.998791 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:43.998920 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.000223 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:44.000226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:44.000228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:44.000248 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:44.001763 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:44.002179 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:44.002185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:44.002189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:44.002280 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:44.002579 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:44.002582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:44.002583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:44.002862 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:44.002969 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.003207 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.003336 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:24:44.016628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44977
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:44.125810 1864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:44.126082 1864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:44.126165 1864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:44.130208 1864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:44.130283 1864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:44.130363 1864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:24:44.130447 1864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:24:44.135574 1864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:44.136579 1864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:44.137647 1864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:44.146987 1869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:44.146988 1872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:44.146988 1870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:44.147305 1864 server_base.cc:1034] running on GCE node
I20260619 02:24:44.147769 1864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:44.148414 1864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:44.149613 1864 hybrid_clock.cc:648] HybridClock initialized: now 1781835884149597 us; error 39 us; skew 500 ppm
Jun 19 02:24:44 dist-test-slave-xkrt krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:44.152359 1864 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:24:44.153184 1864 webserver.cc:460] Webserver started at http://127.0.98.3:45923/ using document root <none> and password file <none>
I20260619 02:24:44.153831 1864 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:44.153919 1864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:44.154145 1864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:44.156194 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "e097fec2ca714f3892a9161127108516"
format_stamp: "Formatted at 2026-06-19 02:24:44 on dist-test-slave-xkrt"
server_key: "fcc47d3c481dd6b79d5f1d3443d8f8bd"
server_key_iv: "89b0f748fbfbfab0493079a96cea95ba"
server_key_version: "encryptionkey@0"
I20260619 02:24:44.156842 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e097fec2ca714f3892a9161127108516"
format_stamp: "Formatted at 2026-06-19 02:24:44 on dist-test-slave-xkrt"
server_key: "fcc47d3c481dd6b79d5f1d3443d8f8bd"
server_key_iv: "89b0f748fbfbfab0493079a96cea95ba"
server_key_version: "encryptionkey@0"
I20260619 02:24:44.160651 1864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:24:44.163203 1879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:44.164484 1864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:24:44.164618 1864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "e097fec2ca714f3892a9161127108516"
format_stamp: "Formatted at 2026-06-19 02:24:44 on dist-test-slave-xkrt"
server_key: "fcc47d3c481dd6b79d5f1d3443d8f8bd"
server_key_iv: "89b0f748fbfbfab0493079a96cea95ba"
server_key_version: "encryptionkey@0"
I20260619 02:24:44.164731 1864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:44.175311 1864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:44.178248 1864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:44.178457 1864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:44.179836 1864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:44.179919 1864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:44.179991 1864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:44.180042 1864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:44.190595 1864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43487
I20260619 02:24:44.190670 1994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43487 every 8 connection(s)
Jun 19 02:24:44 dist-test-slave-xkrt krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:44.191493 1864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:24:44.192185 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 1864
I20260619 02:24:44.192323 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:24:44.192584 392 external_mini_cluster.cc:1428] Setting key d6ee57166237fc9db775371e69f2d297
I20260619 02:24:44.195983 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.180841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39015 (local address 127.0.98.62:44977)
0619 02:24:44.180973 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.180976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.182100 (+ 1124us) server_negotiation.cc:407] Connection header received
0619 02:24:44.183418 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.183421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.183478 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.183568 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:44.185360 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.186269 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.188093 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.188319 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.188448 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.191361 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:44.191365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:44.191368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:44.191400 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:44.194092 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:44.194661 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:44.194664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:44.194666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:44.194778 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:44.195092 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:44.195095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:44.195096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:44.195269 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:44.195344 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.195776 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.195874 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:24:44.196800 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.181342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44977 (local address 127.0.98.3:39015)
0619 02:24:44.181940 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.181981 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.183206 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.183773 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.183787 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.184335 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:44.185170 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.185203 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.186407 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.186411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.187947 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.187961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.188599 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.188601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.188754 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.189354 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:44.189369 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:44.191187 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:44.194197 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:44.194203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:44.194217 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:44.194557 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:44.194869 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:44.194871 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:44.194873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:44.195008 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:44.195354 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:44.195359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:44.195573 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.195723 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.195968 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":169,"threads_started":1}
I20260619 02:24:44.198053 1995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44977
I20260619 02:24:44.198202 1995 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:44.198689 1995 heartbeater.cc:511] Master 127.0.98.62:44977 requested a full tablet report, sending...
I20260619 02:24:44.199770 1512 ts_manager.cc:194] Registered new tserver with Master: e097fec2ca714f3892a9161127108516 (127.0.98.3:43487)
I20260619 02:24:44.200598 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39015
I20260619 02:24:44.206576 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:24:44.207266 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.200277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35675 (local address 127.0.98.62:44977)
0619 02:24:44.200417 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.200421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.200574 (+ 153us) server_negotiation.cc:407] Connection header received
0619 02:24:44.200743 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.200745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.200790 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.200883 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:44.201406 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.201957 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.202922 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.203122 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.203217 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.204230 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:44.204232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:44.204234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:44.204259 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:44.205635 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:44.206082 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:44.206085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:44.206087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:44.206176 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:44.206480 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:44.206482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:44.206483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:44.206664 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:44.206765 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.207024 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.207123 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:24:44.207278 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.200229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44977 (local address 127.0.98.3:35675)
0619 02:24:44.200504 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.200519 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.200620 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.200906 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.200909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.201141 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:44.201259 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.201266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.202083 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.202086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.202813 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.202822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.203232 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.203234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.203302 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.203795 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:44.203809 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:44.204113 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:44.205734 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:44.205736 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:44.205738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:44.205995 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:44.206270 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:44.206273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:44.206275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:44.206372 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:44.206766 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:44.206769 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:44.206865 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.206974 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.207113 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
Jun 19 02:24:44 dist-test-slave-xkrt krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835884, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:24:44 dist-test-slave-xkrt krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835884, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:44.231441 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.223641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.0.98.62:44977)
0619 02:24:44.223750 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.223753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.223763 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:24:44.223796 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.223798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.223832 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.223908 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:44.224558 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.225060 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.225945 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.226166 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.226245 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.228572 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:44.228575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:44.228577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:44.228598 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:44.229973 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:44.230441 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:44.230443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:44.230444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:44.230513 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:44.230733 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:44.230735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:44.230736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:44.230893 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:44.230992 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.231192 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.231303 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:24:44.234212 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:24:44.237287 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:24:44.251631 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.246153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45801 (local address 127.0.0.1:58788)
0619 02:24:44.246406 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.246418 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.246503 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.247551 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.247555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.247573 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:44.247651 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.247656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.248603 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.248609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.250276 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.250286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.251094 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.251098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.251197 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.251232 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.251360 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.251504 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":216,"thread_start_us":101,"threads_started":1}
I20260619 02:24:44.251641 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.246541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.0.98.2:45801)
0619 02:24:44.246865 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.246877 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.246910 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:24:44.246982 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.246989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.247247 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.247414 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:44.247759 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.248488 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.250419 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.250971 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.251094 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.251260 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.251389 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.251487 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":111,"threads_started":1}
I20260619 02:24:44.251907 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.245504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33835 (local address 127.0.0.1:39036)
0619 02:24:44.246066 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.246082 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.246167 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.247073 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.247079 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.247101 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:44.247217 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.247223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.248686 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.248689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.250049 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.250058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.251392 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.251397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.251582 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.251643 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.251709 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.251772 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":190,"thread_start_us":152,"threads_started":1}
I20260619 02:24:44.252115 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.245414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43487 (local address 127.0.0.1:42786)
0619 02:24:44.245880 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.245928 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.246048 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.247379 (+ 1331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.247381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.247397 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:44.247492 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.247499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.248770 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.248773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.250276 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.250286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.251443 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.251447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.251586 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.251644 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.251708 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.251981 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":192,"thread_start_us":101,"threads_started":1}
I20260619 02:24:44.252373 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.245719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.0.98.1:33835)
0619 02:24:44.246415 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.246434 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.246458 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:24:44.246534 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.246539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.246749 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.246907 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:44.247352 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.248553 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.250166 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.251063 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.251176 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.251324 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.252119 (+ 795us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.252237 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":76,"threads_started":1}
I20260619 02:24:44.252949 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.246219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.0.98.3:43487)
0619 02:24:44.246965 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.246975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.246995 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:24:44.247068 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.247072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.247258 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.247405 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:44.247596 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.248672 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.250433 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.251296 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.252338 (+ 1042us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.252682 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.252737 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.252819 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":114,"threads_started":1}
I20260619 02:24:44.254738 1789 tablet_service.cc:1467] Processing CreateTablet for tablet f89f386a89974de3a68f33acb19b9caa (DEFAULT_TABLE table=test-table [id=fa3bd7af12364d0783d1035cf47e9684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:44.254817 1649 tablet_service.cc:1467] Processing CreateTablet for tablet f89f386a89974de3a68f33acb19b9caa (DEFAULT_TABLE table=test-table [id=fa3bd7af12364d0783d1035cf47e9684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:44.255656 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f89f386a89974de3a68f33acb19b9caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:44.255656 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f89f386a89974de3a68f33acb19b9caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:44.262662 2019 tablet_bootstrap.cc:492] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966: Bootstrap starting.
I20260619 02:24:44.263175 1929 tablet_service.cc:1467] Processing CreateTablet for tablet f89f386a89974de3a68f33acb19b9caa (DEFAULT_TABLE table=test-table [id=fa3bd7af12364d0783d1035cf47e9684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:44.264056 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f89f386a89974de3a68f33acb19b9caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:44.264550 2019 tablet_bootstrap.cc:654] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:44.265168 2019 log.cc:824] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:44.265738 2018 tablet_bootstrap.cc:492] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd: Bootstrap starting.
I20260619 02:24:44.267148 2019 tablet_bootstrap.cc:492] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966: No bootstrap required, opened a new log
I20260619 02:24:44.267318 2019 ts_tablet_manager.cc:1397] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 02:24:44.267787 2018 tablet_bootstrap.cc:654] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:44.268570 2018 log.cc:824] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:44.270591 2018 tablet_bootstrap.cc:492] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd: No bootstrap required, opened a new log
I20260619 02:24:44.270823 2018 ts_tablet_manager.cc:1397] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:24:44.270851 2022 tablet_bootstrap.cc:492] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516: Bootstrap starting.
I20260619 02:24:44.271229 2019 raft_consensus.cc:348] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } }
I20260619 02:24:44.271512 2019 raft_consensus.cc:374] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:44.271574 2019 raft_consensus.cc:729] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9319f9f48f8e435a9fba8793032f7966, State: Initialized, Role: FOLLOWER
I20260619 02:24:44.272040 2019 consensus_queue.cc:260] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } }
I20260619 02:24:44.272696 1855 heartbeater.cc:503] Master 127.0.98.62:44977 was elected leader, sending a full tablet report...
I20260619 02:24:44.272897 2022 tablet_bootstrap.cc:654] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:44.273015 2019 ts_tablet_manager.cc:1428] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:24:44.273679 2022 log.cc:824] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:44.275043 2018 raft_consensus.cc:348] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } }
I20260619 02:24:44.275301 2018 raft_consensus.cc:374] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:44.275427 2018 raft_consensus.cc:729] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc9be1d70d814ab4b8ea118e4f144dbd, State: Initialized, Role: FOLLOWER
I20260619 02:24:44.275758 2022 tablet_bootstrap.cc:492] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516: No bootstrap required, opened a new log
I20260619 02:24:44.275842 2018 consensus_queue.cc:260] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } }
I20260619 02:24:44.275938 2022 ts_tablet_manager.cc:1397] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 02:24:44.276345 1715 heartbeater.cc:503] Master 127.0.98.62:44977 was elected leader, sending a full tablet report...
I20260619 02:24:44.276691 2018 ts_tablet_manager.cc:1428] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:24:44.279093 2022 raft_consensus.cc:348] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } }
I20260619 02:24:44.279312 2022 raft_consensus.cc:374] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:44.279374 2022 raft_consensus.cc:729] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e097fec2ca714f3892a9161127108516, State: Initialized, Role: FOLLOWER
I20260619 02:24:44.279721 2022 consensus_queue.cc:260] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } }
I20260619 02:24:44.280314 1995 heartbeater.cc:503] Master 127.0.98.62:44977 was elected leader, sending a full tablet report...
I20260619 02:24:44.280505 2022 ts_tablet_manager.cc:1428] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260619 02:24:44.413666 2025 raft_consensus.cc:482] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:24:44.413872 2025 raft_consensus.cc:504] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } }
I20260619 02:24:44.414958 2025 leader_election.cc:288] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e097fec2ca714f3892a9161127108516 (127.0.98.3:43487), 9319f9f48f8e435a9fba8793032f7966 (127.0.98.2:45801)
I20260619 02:24:44.418952 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.415373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46201 (local address 127.0.98.3:43487)
0619 02:24:44.415508 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.415512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.415643 (+ 131us) server_negotiation.cc:407] Connection header received
0619 02:24:44.415839 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.415842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.415910 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.415953 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:44.416311 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.416814 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.417928 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.418368 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.418484 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.418547 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.418772 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.418839 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:24:44.418972 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.415273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43487 (local address 127.0.98.1:46201)
0619 02:24:44.415541 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.415559 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.415669 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.416072 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.416074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.416089 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:44.416201 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.416205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.416925 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.416930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.417777 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.417786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.418484 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.418488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.418628 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.418647 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.418752 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.418819 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":22,"thread_start_us":111,"threads_started":1}
I20260619 02:24:44.419368 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.415840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38245 (local address 127.0.98.2:45801)
0619 02:24:44.415977 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.415984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.416059 (+ 75us) server_negotiation.cc:407] Connection header received
0619 02:24:44.416255 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.416258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.416311 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.416397 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:44.416694 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.417201 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.418332 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.418814 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.418949 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.419009 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.419206 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.419280 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:24:44.419790 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.415657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45801 (local address 127.0.98.1:38245)
0619 02:24:44.415974 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.415988 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.416129 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.416429 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.416432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.416448 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:44.416575 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.416581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.417315 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.417319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.418206 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.418215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.418950 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.418954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.419051 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.419078 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.419537 (+ 459us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.419653 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":24,"spinlock_wait_cycles":768,"thread_start_us":107,"threads_started":1}
I20260619 02:24:44.419747 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f89f386a89974de3a68f33acb19b9caa" candidate_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e097fec2ca714f3892a9161127108516" is_pre_election: true
I20260619 02:24:44.420058 1949 raft_consensus.cc:2454] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc9be1d70d814ab4b8ea118e4f144dbd in term 0.
I20260619 02:24:44.420468 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f89f386a89974de3a68f33acb19b9caa" candidate_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9319f9f48f8e435a9fba8793032f7966" is_pre_election: true
I20260619 02:24:44.420568 1600 leader_election.cc:302] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e097fec2ca714f3892a9161127108516, fc9be1d70d814ab4b8ea118e4f144dbd; no voters:
I20260619 02:24:44.420747 1809 raft_consensus.cc:2454] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc9be1d70d814ab4b8ea118e4f144dbd in term 0.
I20260619 02:24:44.420878 2025 raft_consensus.cc:2781] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:24:44.421003 2025 raft_consensus.cc:482] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:24:44.421073 2025 raft_consensus.cc:3037] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:44.422339 2025 raft_consensus.cc:504] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } }
I20260619 02:24:44.422672 2025 leader_election.cc:288] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [CANDIDATE]: Term 1 election: Requested vote from peers e097fec2ca714f3892a9161127108516 (127.0.98.3:43487), 9319f9f48f8e435a9fba8793032f7966 (127.0.98.2:45801)
I20260619 02:24:44.423086 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f89f386a89974de3a68f33acb19b9caa" candidate_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9319f9f48f8e435a9fba8793032f7966"
I20260619 02:24:44.423105 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f89f386a89974de3a68f33acb19b9caa" candidate_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e097fec2ca714f3892a9161127108516"
I20260619 02:24:44.423239 1949 raft_consensus.cc:3037] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:44.423287 1809 raft_consensus.cc:3037] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:44.424618 1809 raft_consensus.cc:2454] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc9be1d70d814ab4b8ea118e4f144dbd in term 1.
I20260619 02:24:44.424831 1949 raft_consensus.cc:2454] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc9be1d70d814ab4b8ea118e4f144dbd in term 1.
I20260619 02:24:44.424933 1600 leader_election.cc:302] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9319f9f48f8e435a9fba8793032f7966, fc9be1d70d814ab4b8ea118e4f144dbd; no voters:
I20260619 02:24:44.425102 2025 raft_consensus.cc:2781] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:44.425335 2025 raft_consensus.cc:686] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [term 1 LEADER]: Becoming Leader. State: Replica: fc9be1d70d814ab4b8ea118e4f144dbd, State: Running, Role: LEADER
I20260619 02:24:44.425690 2025 consensus_queue.cc:237] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } }
I20260619 02:24:44.428725 1512 catalog_manager.cc:5495] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd reported cstate change: term changed from 0 to 1, leader changed from <none> to fc9be1d70d814ab4b8ea118e4f144dbd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } health_report { overall_health: UNKNOWN } } }
W20260619 02:24:44.442849 1996 tablet.cc:2369] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:24:44.486915 1856 tablet.cc:2369] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:24:44.490262 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.486074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.0.98.1:33835)
0619 02:24:44.486208 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.486212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.486229 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:24:44.486427 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.486430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.486494 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.486591 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:24:44.486850 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.487661 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.488535 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.488843 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.488959 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.489298 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:24:44.489722 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:24:44.489805 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.490004 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.490053 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:24:44.495915 1809 raft_consensus.cc:1261] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966 [term 1 FOLLOWER]: Refusing update from remote peer fc9be1d70d814ab4b8ea118e4f144dbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:44.495963 1949 raft_consensus.cc:1261] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516 [term 1 FOLLOWER]: Refusing update from remote peer fc9be1d70d814ab4b8ea118e4f144dbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:44.496587 2025 consensus_queue.cc:1035] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:24:44.496837 2029 consensus_queue.cc:1035] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:24:44.505771 2033 mvcc.cc:205] Tried to move back new op lower bound from 7298399782888239104 to 7298399782610198528. Current Snapshot: MvccSnapshot[applied={T|T < 7298399782888239104}]
I20260619 02:24:44.514977 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:24:44.515154 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:24:44.517470 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa: Sending DeleteTablet for 3 replicas of tablet f89f386a89974de3a68f33acb19b9caa
I20260619 02:24:44.518307 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet f89f386a89974de3a68f33acb19b9caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58788
I20260619 02:24:44.518318 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet f89f386a89974de3a68f33acb19b9caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39036
I20260619 02:24:44.518908 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet f89f386a89974de3a68f33acb19b9caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42786
I20260619 02:24:44.518986 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 02:24:44.519376 2041 tablet_replica.cc:330] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd: stopping tablet replica
W20260619 02:24:44.519448 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:24:44.519662 2041 raft_consensus.cc:2229] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:24:44.520058 2041 raft_consensus.cc:2258] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:44.520323 2042 tablet_replica.cc:330] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966: stopping tablet replica
I20260619 02:24:44.520589 2042 raft_consensus.cc:2229] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:24:44.520884 2042 raft_consensus.cc:2258] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:44.521639 2041 ts_tablet_manager.cc:1905] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:44.522217 2042 ts_tablet_manager.cc:1905] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:44.524724 2042 ts_tablet_manager.cc:1918] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:24:44.524736 2041 ts_tablet_manager.cc:1918] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:24:44.524818 2042 log.cc:1196] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/f89f386a89974de3a68f33acb19b9caa
I20260619 02:24:44.524958 2041 log.cc:1196] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/wal/wals/f89f386a89974de3a68f33acb19b9caa
I20260619 02:24:44.525128 2042 ts_tablet_manager.cc:1939] T f89f386a89974de3a68f33acb19b9caa P 9319f9f48f8e435a9fba8793032f7966: Deleting consensus metadata
I20260619 02:24:44.525321 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 1325dae423874614a3bf64a55a26f932 (DEFAULT_TABLE table=test-table [id=dafd6f902db445b6a32c4a2ab70f0f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:44.525480 2041 ts_tablet_manager.cc:1939] T f89f386a89974de3a68f33acb19b9caa P fc9be1d70d814ab4b8ea118e4f144dbd: Deleting consensus metadata
I20260619 02:24:44.525601 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1325dae423874614a3bf64a55a26f932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:44.526273 1496 catalog_manager.cc:4841] TS 9319f9f48f8e435a9fba8793032f7966 (127.0.98.2:45801): tablet f89f386a89974de3a68f33acb19b9caa (table test-table [id=fa3bd7af12364d0783d1035cf47e9684]) successfully deleted
I20260619 02:24:44.526618 2043 tablet_replica.cc:330] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516: stopping tablet replica
I20260619 02:24:44.526909 2043 raft_consensus.cc:2229] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:24:44.527175 2043 raft_consensus.cc:2258] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:44.527693 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 1325dae423874614a3bf64a55a26f932 (DEFAULT_TABLE table=test-table [id=dafd6f902db445b6a32c4a2ab70f0f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:44.527781 2018 tablet_bootstrap.cc:492] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd: Bootstrap starting.
I20260619 02:24:44.527851 1498 catalog_manager.cc:4841] TS fc9be1d70d814ab4b8ea118e4f144dbd (127.0.98.1:33835): tablet f89f386a89974de3a68f33acb19b9caa (table test-table [id=fa3bd7af12364d0783d1035cf47e9684]) successfully deleted
I20260619 02:24:44.527952 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1325dae423874614a3bf64a55a26f932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:44.528482 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 1325dae423874614a3bf64a55a26f932 (DEFAULT_TABLE table=test-table [id=dafd6f902db445b6a32c4a2ab70f0f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:44.528728 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1325dae423874614a3bf64a55a26f932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:44.528728 2018 tablet_bootstrap.cc:654] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:44.530184 2022 tablet_bootstrap.cc:492] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516: Bootstrap starting.
I20260619 02:24:44.531340 2018 tablet_bootstrap.cc:492] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd: No bootstrap required, opened a new log
I20260619 02:24:44.531410 2018 ts_tablet_manager.cc:1397] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 02:24:44.531569 2022 tablet_bootstrap.cc:654] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:44.531569 2019 tablet_bootstrap.cc:492] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966: Bootstrap starting.
I20260619 02:24:44.531813 2018 raft_consensus.cc:348] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.531929 2018 raft_consensus.cc:374] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:44.531960 2018 raft_consensus.cc:729] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc9be1d70d814ab4b8ea118e4f144dbd, State: Initialized, Role: FOLLOWER
I20260619 02:24:44.532066 2018 consensus_queue.cc:260] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.532297 2018 ts_tablet_manager.cc:1428] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:24:44.532464 2019 tablet_bootstrap.cc:654] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:44.533609 2043 ts_tablet_manager.cc:1905] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:44.534464 2019 tablet_bootstrap.cc:492] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966: No bootstrap required, opened a new log
I20260619 02:24:44.534547 2019 ts_tablet_manager.cc:1397] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:24:44.534982 2019 raft_consensus.cc:348] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.535061 2019 raft_consensus.cc:374] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:44.535082 2019 raft_consensus.cc:729] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9319f9f48f8e435a9fba8793032f7966, State: Initialized, Role: FOLLOWER
I20260619 02:24:44.535209 2019 consensus_queue.cc:260] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.535492 2019 ts_tablet_manager.cc:1428] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:24:44.535887 2043 ts_tablet_manager.cc:1918] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:24:44.535980 2043 log.cc:1196] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/f89f386a89974de3a68f33acb19b9caa
I20260619 02:24:44.536273 2043 ts_tablet_manager.cc:1939] T f89f386a89974de3a68f33acb19b9caa P e097fec2ca714f3892a9161127108516: Deleting consensus metadata
I20260619 02:24:44.536494 2022 tablet_bootstrap.cc:492] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516: No bootstrap required, opened a new log
I20260619 02:24:44.536571 2022 ts_tablet_manager.cc:1397] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260619 02:24:44.536998 2022 raft_consensus.cc:348] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.537078 2022 raft_consensus.cc:374] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:44.537122 2022 raft_consensus.cc:729] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e097fec2ca714f3892a9161127108516, State: Initialized, Role: FOLLOWER
I20260619 02:24:44.537388 1496 catalog_manager.cc:4841] TS e097fec2ca714f3892a9161127108516 (127.0.98.3:43487): tablet f89f386a89974de3a68f33acb19b9caa (table test-table [id=fa3bd7af12364d0783d1035cf47e9684]) successfully deleted
I20260619 02:24:44.537380 2022 consensus_queue.cc:260] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.537906 2022 ts_tablet_manager.cc:1428] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260619 02:24:44.568763 2026 raft_consensus.cc:482] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:24:44.568920 2026 raft_consensus.cc:504] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.570153 2026 leader_election.cc:288] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc9be1d70d814ab4b8ea118e4f144dbd (127.0.98.1:33835), 9319f9f48f8e435a9fba8793032f7966 (127.0.98.2:45801)
I20260619 02:24:44.574447 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.570315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33835 (local address 127.0.98.3:39045)
0619 02:24:44.570429 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.570445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.570541 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.571024 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.571027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.571039 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:44.571157 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.571163 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.571941 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.571948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.573077 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.573086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.574039 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.574042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.574143 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.574168 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.574243 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.574313 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 02:24:44.574472 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.570694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39045 (local address 127.0.98.1:33835)
0619 02:24:44.570802 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.570806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.570817 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:24:44.570884 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.570887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.570936 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.571016 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:44.571244 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.571827 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.573219 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.573875 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.573981 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.574045 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.574278 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.574348 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:24:44.575140 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.570380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45801 (local address 127.0.98.3:39197)
0619 02:24:44.570776 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.570790 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.570895 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.571156 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.571160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.571182 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:44.571308 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.571315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.572349 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.572352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.573867 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.573875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.574725 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.574728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.574831 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.574848 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.574940 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.575008 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":77,"threads_started":1}
I20260619 02:24:44.575129 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325dae423874614a3bf64a55a26f932" candidate_uuid: "e097fec2ca714f3892a9161127108516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" is_pre_election: true
I20260619 02:24:44.575394 1669 raft_consensus.cc:2454] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e097fec2ca714f3892a9161127108516 in term 0.
I20260619 02:24:44.575737 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.570564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39197 (local address 127.0.98.2:45801)
0619 02:24:44.570684 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.570688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.570883 (+ 195us) server_negotiation.cc:407] Connection header received
0619 02:24:44.570980 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.570984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.571034 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.571098 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:44.571449 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.572231 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.573999 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.574617 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.574718 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.574774 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.575523 (+ 749us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.575611 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:24:44.575836 1882 leader_election.cc:302] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e097fec2ca714f3892a9161127108516, fc9be1d70d814ab4b8ea118e4f144dbd; no voters:
I20260619 02:24:44.576093 2026 raft_consensus.cc:2781] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:24:44.576175 2026 raft_consensus.cc:482] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:24:44.576146 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325dae423874614a3bf64a55a26f932" candidate_uuid: "e097fec2ca714f3892a9161127108516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9319f9f48f8e435a9fba8793032f7966" is_pre_election: true
I20260619 02:24:44.576223 2026 raft_consensus.cc:3037] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:44.576272 1809 raft_consensus.cc:2454] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e097fec2ca714f3892a9161127108516 in term 0.
I20260619 02:24:44.577450 2026 raft_consensus.cc:504] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.577888 2026 leader_election.cc:288] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [CANDIDATE]: Term 1 election: Requested vote from peers fc9be1d70d814ab4b8ea118e4f144dbd (127.0.98.1:33835), 9319f9f48f8e435a9fba8793032f7966 (127.0.98.2:45801)
I20260619 02:24:44.578162 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325dae423874614a3bf64a55a26f932" candidate_uuid: "e097fec2ca714f3892a9161127108516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd"
I20260619 02:24:44.578197 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325dae423874614a3bf64a55a26f932" candidate_uuid: "e097fec2ca714f3892a9161127108516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9319f9f48f8e435a9fba8793032f7966"
I20260619 02:24:44.578284 1669 raft_consensus.cc:3037] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:44.578341 1809 raft_consensus.cc:3037] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:44.579223 1809 raft_consensus.cc:2454] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e097fec2ca714f3892a9161127108516 in term 1.
I20260619 02:24:44.579423 1669 raft_consensus.cc:2454] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e097fec2ca714f3892a9161127108516 in term 1.
I20260619 02:24:44.579533 1880 leader_election.cc:302] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9319f9f48f8e435a9fba8793032f7966, e097fec2ca714f3892a9161127108516; no voters:
I20260619 02:24:44.579763 2026 raft_consensus.cc:2781] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:44.580017 2026 raft_consensus.cc:686] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [term 1 LEADER]: Becoming Leader. State: Replica: e097fec2ca714f3892a9161127108516, State: Running, Role: LEADER
I20260619 02:24:44.580264 2026 consensus_queue.cc:237] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.582394 1512 catalog_manager.cc:5495] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 reported cstate change: term changed from 0 to 1, leader changed from <none> to e097fec2ca714f3892a9161127108516 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e097fec2ca714f3892a9161127108516" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } health_report { overall_health: HEALTHY } } }
Jun 19 02:24:44 dist-test-slave-xkrt krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:44.632661 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.623290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44977 (local address 127.0.0.1:42832)
0619 02:24:44.623626 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.623637 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.623739 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.624017 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.624019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.624369 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:44.624472 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.624477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.625523 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.625525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.626132 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.626139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.626679 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.626681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.626772 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.627356 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:44.627368 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:44.629006 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:44.630953 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:44.630961 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:44.630965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:44.631182 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:44.631453 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:44.631455 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:44.631457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:44.631530 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:44.632217 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:44.632222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:44.632316 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.632415 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.632526 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":18,"thread_start_us":105,"threads_started":1}
I20260619 02:24:44.632781 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.623360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42832 (local address 127.0.98.62:44977)
0619 02:24:44.623493 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.623497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.623729 (+ 232us) server_negotiation.cc:407] Connection header received
0619 02:24:44.623821 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.623834 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.623896 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.623987 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:44.624598 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.625412 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.626248 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.626570 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.626682 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.629135 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:44.629140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:44.629145 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:44.629171 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:44.630855 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:44.631272 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:44.631276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:44.631278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:44.631365 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:44.631899 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:44.631902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:44.631904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:44.632113 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:44.632215 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.632488 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.632604 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":56}
I20260619 02:24:44.636668 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42832:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:24:44.637396 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:24:44.642278 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 49855fce3b054d2cb6b7a080c620d237 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ae2cca087204fdaad9f84f07e0ac66a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:24:44.642375 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 49855fce3b054d2cb6b7a080c620d237 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ae2cca087204fdaad9f84f07e0ac66a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:24:44.642458 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 49855fce3b054d2cb6b7a080c620d237 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ae2cca087204fdaad9f84f07e0ac66a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:24:44.642704 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49855fce3b054d2cb6b7a080c620d237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:44.642704 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49855fce3b054d2cb6b7a080c620d237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:44.642843 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49855fce3b054d2cb6b7a080c620d237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:44.645506 2018 tablet_bootstrap.cc:492] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd: Bootstrap starting.
I20260619 02:24:44.645601 2019 tablet_bootstrap.cc:492] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966: Bootstrap starting.
I20260619 02:24:44.646543 2019 tablet_bootstrap.cc:654] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:44.646579 2018 tablet_bootstrap.cc:654] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:44.647835 2019 tablet_bootstrap.cc:492] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966: No bootstrap required, opened a new log
I20260619 02:24:44.647858 2022 tablet_bootstrap.cc:492] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516: Bootstrap starting.
I20260619 02:24:44.647934 2019 ts_tablet_manager.cc:1397] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:24:44.648371 2019 raft_consensus.cc:348] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.648483 2019 raft_consensus.cc:374] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:44.648514 2019 raft_consensus.cc:729] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9319f9f48f8e435a9fba8793032f7966, State: Initialized, Role: FOLLOWER
I20260619 02:24:44.648698 2018 tablet_bootstrap.cc:492] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd: No bootstrap required, opened a new log
I20260619 02:24:44.648629 2019 consensus_queue.cc:260] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.648777 2022 tablet_bootstrap.cc:654] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:44.648777 2018 ts_tablet_manager.cc:1397] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 02:24:44.648954 2019 ts_tablet_manager.cc:1428] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:24:44.649209 2018 raft_consensus.cc:348] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.649326 2018 raft_consensus.cc:374] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:44.649363 2018 raft_consensus.cc:729] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc9be1d70d814ab4b8ea118e4f144dbd, State: Initialized, Role: FOLLOWER
I20260619 02:24:44.649482 2018 consensus_queue.cc:260] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.649756 2018 ts_tablet_manager.cc:1428] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:24:44.649876 2025 tablet_replica.cc:439] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } }
I20260619 02:24:44.650009 2025 tablet_replica.cc:442] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:24:44.650110 2023 tablet_replica.cc:439] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } }
I20260619 02:24:44.650219 2023 tablet_replica.cc:442] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:24:44.651436 2022 tablet_bootstrap.cc:492] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516: No bootstrap required, opened a new log
I20260619 02:24:44.651535 2022 ts_tablet_manager.cc:1397] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 02:24:44.652004 2022 raft_consensus.cc:348] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.652298 2022 raft_consensus.cc:374] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:44.652348 2022 raft_consensus.cc:729] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e097fec2ca714f3892a9161127108516, State: Initialized, Role: FOLLOWER
I20260619 02:24:44.652480 2022 consensus_queue.cc:260] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.652714 2026 tablet_replica.cc:439] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } }
I20260619 02:24:44.652798 2022 ts_tablet_manager.cc:1428] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:24:44.652827 2026 tablet_replica.cc:442] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 02:24:44.759147 1716 tablet.cc:2369] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:24:44.768905 2023 raft_consensus.cc:482] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:24:44.769081 2023 raft_consensus.cc:504] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.770368 2023 leader_election.cc:288] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc9be1d70d814ab4b8ea118e4f144dbd (127.0.98.1:33835), e097fec2ca714f3892a9161127108516 (127.0.98.3:43487)
I20260619 02:24:44.774716 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.770616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43487 (local address 127.0.98.2:34389)
0619 02:24:44.771090 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.771111 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.771233 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.771557 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.771560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.771576 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:44.771705 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.771712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.772690 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.772695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.773490 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.773498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.774238 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.774242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.774361 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.774389 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.774472 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.774545 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":104,"threads_started":1}
I20260619 02:24:44.774722 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.770873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34389 (local address 127.0.98.3:43487)
0619 02:24:44.770986 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.770991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.771212 (+ 221us) server_negotiation.cc:407] Connection header received
0619 02:24:44.771322 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.771325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.771373 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.771442 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:44.771843 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.772581 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.773655 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.774117 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.774219 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.774280 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.774505 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.774575 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:24:44.775239 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.770553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33835 (local address 127.0.98.2:58971)
0619 02:24:44.770955 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.770971 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.771075 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.771350 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.771353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.771366 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:44.771468 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.771474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.772455 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.772458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.773911 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.773921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.774805 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.774809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.774907 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.774942 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.775027 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.775093 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":109,"threads_started":1}
I20260619 02:24:44.775254 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.770776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58971 (local address 127.0.98.1:33835)
0619 02:24:44.770906 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.770911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.771038 (+ 127us) server_negotiation.cc:407] Connection header received
0619 02:24:44.771198 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.771202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.771249 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.771325 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:44.771584 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.772321 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.774041 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.774590 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.774685 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.774739 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.775062 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.775146 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:24:44.775439 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49855fce3b054d2cb6b7a080c620d237" candidate_uuid: "9319f9f48f8e435a9fba8793032f7966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e097fec2ca714f3892a9161127108516" is_pre_election: true
I20260619 02:24:44.775575 1949 raft_consensus.cc:2454] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9319f9f48f8e435a9fba8793032f7966 in term 0.
I20260619 02:24:44.775791 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49855fce3b054d2cb6b7a080c620d237" candidate_uuid: "9319f9f48f8e435a9fba8793032f7966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" is_pre_election: true
I20260619 02:24:44.775942 1669 raft_consensus.cc:2454] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9319f9f48f8e435a9fba8793032f7966 in term 0.
I20260619 02:24:44.776002 1740 leader_election.cc:302] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9319f9f48f8e435a9fba8793032f7966, e097fec2ca714f3892a9161127108516; no voters:
I20260619 02:24:44.776270 2023 raft_consensus.cc:2781] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:24:44.776358 2023 raft_consensus.cc:482] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:24:44.776414 2023 raft_consensus.cc:3037] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:44.777364 2023 raft_consensus.cc:504] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.777724 2023 leader_election.cc:288] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [CANDIDATE]: Term 1 election: Requested vote from peers fc9be1d70d814ab4b8ea118e4f144dbd (127.0.98.1:33835), e097fec2ca714f3892a9161127108516 (127.0.98.3:43487)
I20260619 02:24:44.778119 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49855fce3b054d2cb6b7a080c620d237" candidate_uuid: "9319f9f48f8e435a9fba8793032f7966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e097fec2ca714f3892a9161127108516"
I20260619 02:24:44.778223 1949 raft_consensus.cc:3037] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:44.778384 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49855fce3b054d2cb6b7a080c620d237" candidate_uuid: "9319f9f48f8e435a9fba8793032f7966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd"
I20260619 02:24:44.778519 1669 raft_consensus.cc:3037] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:44.779412 1949 raft_consensus.cc:2454] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9319f9f48f8e435a9fba8793032f7966 in term 1.
I20260619 02:24:44.779675 1669 raft_consensus.cc:2454] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9319f9f48f8e435a9fba8793032f7966 in term 1.
I20260619 02:24:44.779702 1740 leader_election.cc:302] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9319f9f48f8e435a9fba8793032f7966, e097fec2ca714f3892a9161127108516; no voters:
I20260619 02:24:44.779861 2023 raft_consensus.cc:2781] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:44.780055 2023 raft_consensus.cc:686] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [term 1 LEADER]: Becoming Leader. State: Replica: 9319f9f48f8e435a9fba8793032f7966, State: Running, Role: LEADER
I20260619 02:24:44.780349 2023 consensus_queue.cc:237] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } }
I20260619 02:24:44.781430 2061 tablet_replica.cc:439] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966: TxnStatusTablet state changed. Reason: New leader 9319f9f48f8e435a9fba8793032f7966. Latest consensus state: current_term: 1 leader_uuid: "9319f9f48f8e435a9fba8793032f7966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } }
I20260619 02:24:44.781574 2061 tablet_replica.cc:442] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:24:44.781845 2063 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:24:44.782024 2063 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:24:44.782735 1511 catalog_manager.cc:5495] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9319f9f48f8e435a9fba8793032f7966 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9319f9f48f8e435a9fba8793032f7966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } health_report { overall_health: UNKNOWN } } }
I20260619 02:24:44.809070 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.805017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45801 (local address 127.0.0.1:49478)
0619 02:24:44.805192 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.805205 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.805292 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.805696 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.805698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.805706 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:24:44.805798 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.805802 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.806802 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.806804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.807405 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.807412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.807978 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.807981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.808044 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.808085 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:24:44.808830 (+ 745us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:24:44.808841 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.808889 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.808961 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 02:24:44.809124 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.805105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.0.98.2:45801)
0619 02:24:44.805248 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.805253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.805269 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:24:44.805446 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.805449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.805512 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.805662 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:24:44.805915 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.806694 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.807514 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.807861 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.807989 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.808217 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:24:44.808735 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:24:44.808845 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.808921 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.808974 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:24:44.811826 1949 raft_consensus.cc:1261] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516 [term 1 FOLLOWER]: Refusing update from remote peer 9319f9f48f8e435a9fba8793032f7966: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:44.811985 1669 raft_consensus.cc:1261] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd [term 1 FOLLOWER]: Refusing update from remote peer 9319f9f48f8e435a9fba8793032f7966: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:44.812371 2061 consensus_queue.cc:1035] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:24:44.812536 2023 consensus_queue.cc:1035] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:24:44.814219 2026 tablet_replica.cc:439] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516: TxnStatusTablet state changed. Reason: New leader 9319f9f48f8e435a9fba8793032f7966. Latest consensus state: current_term: 1 leader_uuid: "9319f9f48f8e435a9fba8793032f7966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } }
I20260619 02:24:44.814347 2026 tablet_replica.cc:442] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:24:44.814311 2025 tablet_replica.cc:439] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd: TxnStatusTablet state changed. Reason: New leader 9319f9f48f8e435a9fba8793032f7966. Latest consensus state: current_term: 1 leader_uuid: "9319f9f48f8e435a9fba8793032f7966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } }
I20260619 02:24:44.814410 2025 tablet_replica.cc:442] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:24:44.815717 2026 tablet_replica.cc:439] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9319f9f48f8e435a9fba8793032f7966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } }
I20260619 02:24:44.815822 2026 tablet_replica.cc:442] T 49855fce3b054d2cb6b7a080c620d237 P e097fec2ca714f3892a9161127108516: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:24:44.816067 2023 tablet_replica.cc:439] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9319f9f48f8e435a9fba8793032f7966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } }
I20260619 02:24:44.816179 2023 tablet_replica.cc:442] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:24:44.816291 2034 mvcc.cc:205] Tried to move back new op lower bound from 7298399784184045568 to 7298399784062418944. Current Snapshot: MvccSnapshot[applied={T|T < 7298399784184045568 or (T in {7298399784184045568})}]
I20260619 02:24:44.817104 2025 tablet_replica.cc:439] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9319f9f48f8e435a9fba8793032f7966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } }
I20260619 02:24:44.817204 2025 tablet_replica.cc:442] T 49855fce3b054d2cb6b7a080c620d237 P fc9be1d70d814ab4b8ea118e4f144dbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:24:44.817243 2032 mvcc.cc:205] Tried to move back new op lower bound from 7298399784184045568 to 7298399784062418944. Current Snapshot: MvccSnapshot[applied={T|T < 7298399784184045568}]
I20260619 02:24:44.818011 2061 tablet_replica.cc:439] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9319f9f48f8e435a9fba8793032f7966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 } } peers { permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 } } peers { permanent_uuid: "e097fec2ca714f3892a9161127108516" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43487 } } }
I20260619 02:24:44.818491 2061 tablet_replica.cc:442] T 49855fce3b054d2cb6b7a080c620d237 P 9319f9f48f8e435a9fba8793032f7966: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:24:44.825425 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.821481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.0.98.3:43487)
0619 02:24:44.821874 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.821877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.821887 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:24:44.821941 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.821944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.821986 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.822095 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:24:44.822336 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.822938 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.824099 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.824301 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.824371 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.824637 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:24:44.825033 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:24:44.825118 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.825225 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.825268 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318}
I20260619 02:24:44.835317 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.831884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42255 (local address 127.0.98.2:45801)
0619 02:24:44.832016 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.832022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.832189 (+ 167us) server_negotiation.cc:407] Connection header received
0619 02:24:44.832310 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.832312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.832363 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.832422 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:24:44.832660 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.833125 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.834361 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.834568 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.834643 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.834951 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:24:44.835010 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:24:44.835061 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.835177 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.835218 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:24:44.835384 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.831800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45801 (local address 127.0.98.3:42255)
0619 02:24:44.832105 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.832120 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.832220 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.832453 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.832456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.832464 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:24:44.832566 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.832571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.833220 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.833223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.834243 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.834253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.834686 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.834690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.834779 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.834826 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:24:44.835087 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:24:44.835093 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.835158 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.835232 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":103,"threads_started":1}
I20260619 02:24:44.841912 1809 raft_consensus.cc:1261] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966 [term 1 FOLLOWER]: Refusing update from remote peer e097fec2ca714f3892a9161127108516: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:44.842236 1669 raft_consensus.cc:1261] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd [term 1 FOLLOWER]: Refusing update from remote peer e097fec2ca714f3892a9161127108516: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:44.842449 2048 consensus_queue.cc:1035] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9319f9f48f8e435a9fba8793032f7966" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:24:44.842602 2026 consensus_queue.cc:1035] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc9be1d70d814ab4b8ea118e4f144dbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:24:44.864880 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.860949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48417 (local address 127.0.98.3:43487)
0619 02:24:44.861070 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.861075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.861273 (+ 198us) server_negotiation.cc:407] Connection header received
0619 02:24:44.861386 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.861388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.861425 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.861556 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:24:44.861828 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.862604 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.863877 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.864084 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.864160 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.864460 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:24:44.864519 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:24:44.864596 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.864695 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.864744 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:24:44.864915 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.860881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43487 (local address 127.0.98.2:48417)
0619 02:24:44.861181 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:44.861196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:44.861299 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:44.861579 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:44.861582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:44.861590 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:24:44.861705 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.861711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.862741 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.862744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.863748 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:44.863762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.864188 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.864192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:44.864291 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.864341 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:24:44.864608 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:24:44.864614 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:24:44.864673 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 02:24:44.864748 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":110,"threads_started":1}
I20260619 02:24:44.889946 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:24:44.890110 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:24:44.892251 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd2b3115c9084bf0807e632b291296aa: Sending DeleteTablet for 3 replicas of tablet 1325dae423874614a3bf64a55a26f932
I20260619 02:24:44.892748 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet 1325dae423874614a3bf64a55a26f932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39036
I20260619 02:24:44.892956 2041 tablet_replica.cc:330] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd: stopping tablet replica
I20260619 02:24:44.893102 2041 raft_consensus.cc:2229] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:24:44.893087 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet 1325dae423874614a3bf64a55a26f932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58788
I20260619 02:24:44.893115 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 1325dae423874614a3bf64a55a26f932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42786
I20260619 02:24:44.893227 2041 raft_consensus.cc:2258] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:44.893285 2042 tablet_replica.cc:330] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966: stopping tablet replica
I20260619 02:24:44.893399 2042 raft_consensus.cc:2229] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:24:44.893499 2042 raft_consensus.cc:2258] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:44.893755 2043 tablet_replica.cc:330] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516: stopping tablet replica
I20260619 02:24:44.893877 2043 raft_consensus.cc:2229] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:24:44.894153 2043 raft_consensus.cc:2258] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:44.894201 2041 ts_tablet_manager.cc:1905] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:44.894258 2042 ts_tablet_manager.cc:1905] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:44.896725 2041 ts_tablet_manager.cc:1918] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:24:44.896823 2041 log.cc:1196] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-0/wal/wals/1325dae423874614a3bf64a55a26f932
I20260619 02:24:44.897055 2041 ts_tablet_manager.cc:1939] T 1325dae423874614a3bf64a55a26f932 P fc9be1d70d814ab4b8ea118e4f144dbd: Deleting consensus metadata
I20260619 02:24:44.897049 2042 ts_tablet_manager.cc:1918] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:24:44.897127 2042 log.cc:1196] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/1325dae423874614a3bf64a55a26f932
I20260619 02:24:44.897847 2043 ts_tablet_manager.cc:1905] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:44.898207 1498 catalog_manager.cc:4841] TS fc9be1d70d814ab4b8ea118e4f144dbd (127.0.98.1:33835): tablet 1325dae423874614a3bf64a55a26f932 (table test-table [id=dafd6f902db445b6a32c4a2ab70f0f11]) successfully deleted
I20260619 02:24:44.898576 2042 ts_tablet_manager.cc:1939] T 1325dae423874614a3bf64a55a26f932 P 9319f9f48f8e435a9fba8793032f7966: Deleting consensus metadata
I20260619 02:24:44.899223 1496 catalog_manager.cc:4841] TS 9319f9f48f8e435a9fba8793032f7966 (127.0.98.2:45801): tablet 1325dae423874614a3bf64a55a26f932 (table test-table [id=dafd6f902db445b6a32c4a2ab70f0f11]) successfully deleted
I20260619 02:24:44.900449 2043 ts_tablet_manager.cc:1918] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:24:44.900514 2043 log.cc:1196] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/1325dae423874614a3bf64a55a26f932
I20260619 02:24:44.900741 2043 ts_tablet_manager.cc:1939] T 1325dae423874614a3bf64a55a26f932 P e097fec2ca714f3892a9161127108516: Deleting consensus metadata
I20260619 02:24:44.901520 1496 catalog_manager.cc:4841] TS e097fec2ca714f3892a9161127108516 (127.0.98.3:43487): tablet 1325dae423874614a3bf64a55a26f932 (table test-table [id=dafd6f902db445b6a32c4a2ab70f0f11]) successfully deleted
Jun 19 02:24:44 dist-test-slave-xkrt krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835884, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 02:24:44.912240 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.899684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52350 (local address 127.0.98.1:33835)
0619 02:24:44.899786 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.899790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.900364 (+ 574us) server_negotiation.cc:407] Connection header received
0619 02:24:44.900581 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.900584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.900630 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.900679 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:44.901691 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.902483 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.903788 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.904117 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.904262 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.907993 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:44.908004 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:44.908020 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:44.908048 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:44.910091 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:44.910580 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:44.910586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:44.910588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:44.910676 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:44.911262 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:44.911265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:44.911267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:44.911546 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:44.911747 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.911955 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.912075 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 02:24:44.912618 1709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52350
I20260619 02:24:44.923856 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:44.914494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.0.98.62:44977)
0619 02:24:44.914649 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:44.914654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:44.914960 (+ 306us) server_negotiation.cc:407] Connection header received
0619 02:24:44.915066 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:44.915069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:44.915120 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:44.915199 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:44.915928 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.916767 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.917936 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:44.918293 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:44.918397 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:44.920219 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:44.920224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:44.920228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:44.920259 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:44.922079 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:44.922583 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:44.922588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:44.922590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:44.922678 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:44.922965 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:44.922968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:44.922970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:44.923200 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:44.923336 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:44.923577 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 02:24:44.923690 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 02:24:44.924340 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42836
I20260619 02:24:44.925835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 1584
I20260619 02:24:44.938211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 1724
I20260619 02:24:44.945225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 1864
I20260619 02:24:44.951108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 1480
2026-06-19T02:24:44Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3845 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:24:45 dist-test-slave-xkrt krb5kdc[2085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:24:45 dist-test-slave-xkrt krb5kdc[2085](info): set up 2 sockets
Jun 19 02:24:45 dist-test-slave-xkrt krb5kdc[2085](info): commencing operation
krb5kdc: starting...
W20260619 02:24:46.009305 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:24:46 dist-test-slave-xkrt krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:24:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:24:46.130371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43821
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:46.236809 2098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:46.237056 2098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:46.237095 2098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:46.240938 2098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:24:46.240988 2098 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:24:46.241004 2098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:46.241021 2098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:46.241037 2098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:24:46.241051 2098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:24:46.246088 2098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43821
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:46.247017 2098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:46.247932 2098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:46.256083 2103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:46.256093 2104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:46.256094 2106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:46.256143 2098 server_base.cc:1034] running on GCE node
I20260619 02:24:46.256837 2098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:46.257988 2098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:46.259174 2098 hybrid_clock.cc:648] HybridClock initialized: now 1781835886259166 us; error 27 us; skew 500 ppm
Jun 19 02:24:46 dist-test-slave-xkrt krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:46.261873 2098 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:24:46.262735 2098 webserver.cc:460] Webserver started at http://127.0.98.62:42669/ using document root <none> and password file <none>
I20260619 02:24:46.263276 2098 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:46.263393 2098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:46.263622 2098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:46.265615 2098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "af1e68eda2c7480d9d1b1bcc2ce214f2"
format_stamp: "Formatted at 2026-06-19 02:24:46 on dist-test-slave-xkrt"
server_key: "049e00a02902a24064b6d8100b53b938"
server_key_iv: "81da1bcd963caca68af195f1d60ec8a5"
server_key_version: "encryptionkey@0"
I20260619 02:24:46.266175 2098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "af1e68eda2c7480d9d1b1bcc2ce214f2"
format_stamp: "Formatted at 2026-06-19 02:24:46 on dist-test-slave-xkrt"
server_key: "049e00a02902a24064b6d8100b53b938"
server_key_iv: "81da1bcd963caca68af195f1d60ec8a5"
server_key_version: "encryptionkey@0"
I20260619 02:24:46.269718 2098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:24:46.272061 2113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:46.273177 2098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:24:46.273312 2098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "af1e68eda2c7480d9d1b1bcc2ce214f2"
format_stamp: "Formatted at 2026-06-19 02:24:46 on dist-test-slave-xkrt"
server_key: "049e00a02902a24064b6d8100b53b938"
server_key_iv: "81da1bcd963caca68af195f1d60ec8a5"
server_key_version: "encryptionkey@0"
I20260619 02:24:46.273432 2098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:46.286327 2098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:46.289645 2098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:46.289842 2098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:46.297430 2098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43821
I20260619 02:24:46.297441 2175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43821 every 8 connection(s)
I20260619 02:24:46.298393 2098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:24:46.301448 2176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:46.305868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 2098
I20260619 02:24:46.305994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:24:46.306275 392 external_mini_cluster.cc:1428] Setting key 2eb42a8a0328886a4e9cf23a21799312
I20260619 02:24:46.307232 2176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2: Bootstrap starting.
I20260619 02:24:46.309415 2176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:46.310317 2176 log.cc:824] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:46.312300 2176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2: No bootstrap required, opened a new log
Jun 19 02:24:46 dist-test-slave-xkrt krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:46.316218 2176 raft_consensus.cc:348] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1e68eda2c7480d9d1b1bcc2ce214f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43821 } }
I20260619 02:24:46.316428 2176 raft_consensus.cc:374] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:46.316509 2176 raft_consensus.cc:729] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1e68eda2c7480d9d1b1bcc2ce214f2, State: Initialized, Role: FOLLOWER
I20260619 02:24:46.317042 2176 consensus_queue.cc:260] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1e68eda2c7480d9d1b1bcc2ce214f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43821 } }
I20260619 02:24:46.317195 2176 raft_consensus.cc:388] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:24:46.317272 2176 raft_consensus.cc:482] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:24:46.317368 2176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:46.318429 2176 raft_consensus.cc:504] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1e68eda2c7480d9d1b1bcc2ce214f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43821 } }
I20260619 02:24:46.318783 2176 leader_election.cc:302] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af1e68eda2c7480d9d1b1bcc2ce214f2; no voters:
I20260619 02:24:46.319097 2176 leader_election.cc:288] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:24:46.319195 2181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:46.319445 2181 raft_consensus.cc:686] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [term 1 LEADER]: Becoming Leader. State: Replica: af1e68eda2c7480d9d1b1bcc2ce214f2, State: Running, Role: LEADER
I20260619 02:24:46.319913 2181 consensus_queue.cc:237] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1e68eda2c7480d9d1b1bcc2ce214f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43821 } }
I20260619 02:24:46.320408 2176 sys_catalog.cc:564] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:24:46.320739 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:46.307606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.0.98.62:43821)
0619 02:24:46.308005 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:46.308020 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:46.308052 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:24:46.308645 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:46.308661 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:46.308979 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:46.309244 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:46.310140 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.311129 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.312240 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.312737 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.312855 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:46.315591 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:46.315596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:46.315608 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:46.315633 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:46.317702 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:46.318110 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:46.318117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:46.318123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:46.318264 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:46.318529 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:46.318533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:46.318534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:46.319047 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:46.319235 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:46.319562 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 02:24:46.319842 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":125,"threads_started":1}
I20260619 02:24:46.320969 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af1e68eda2c7480d9d1b1bcc2ce214f2. Latest consensus state: current_term: 1 leader_uuid: "af1e68eda2c7480d9d1b1bcc2ce214f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1e68eda2c7480d9d1b1bcc2ce214f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43821 } } }
I20260619 02:24:46.321084 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:46.321260 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af1e68eda2c7480d9d1b1bcc2ce214f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1e68eda2c7480d9d1b1bcc2ce214f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43821 } } }
I20260619 02:24:46.321409 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2 [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:46.321805 2185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:24:46.324887 2185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:24:46.330063 2185 catalog_manager.cc:1347] Generated new cluster ID: ebe3053757d24c2f87ab4ba2267cab7d
I20260619 02:24:46.330127 2185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:24:46.342782 2185 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:24:46.343796 2185 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:24:46.351610 2185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2: Generated new TSK 0
I20260619 02:24:46.352357 2185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:24:46.389168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43821
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:46.498805 2202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:46.499055 2202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:46.499102 2202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:46.503036 2202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:46.503085 2202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:46.503123 2202 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:24:46.503160 2202 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:24:46.503268 2202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:24:46.508396 2202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:46.509382 2202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:46.510346 2202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:46.519611 2207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:46.519640 2210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:46.519834 2202 server_base.cc:1034] running on GCE node
W20260619 02:24:46.519640 2208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:46.520342 2202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:46.520941 2202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:46.522132 2202 hybrid_clock.cc:648] HybridClock initialized: now 1781835886522061 us; error 84 us; skew 500 ppm
Jun 19 02:24:46 dist-test-slave-xkrt krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:46.524693 2202 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:24:46.525471 2202 webserver.cc:460] Webserver started at http://127.0.98.1:40747/ using document root <none> and password file <none>
I20260619 02:24:46.526026 2202 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:46.526115 2202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:46.526332 2202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:46.528209 2202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "02258e2a40a346ea80523911c8542b20"
format_stamp: "Formatted at 2026-06-19 02:24:46 on dist-test-slave-xkrt"
server_key: "db42e45e1a3e589209e5ff56ba81cdbf"
server_key_iv: "4ee7185de96001c479475ab094df58b6"
server_key_version: "encryptionkey@0"
I20260619 02:24:46.528764 2202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "02258e2a40a346ea80523911c8542b20"
format_stamp: "Formatted at 2026-06-19 02:24:46 on dist-test-slave-xkrt"
server_key: "db42e45e1a3e589209e5ff56ba81cdbf"
server_key_iv: "4ee7185de96001c479475ab094df58b6"
server_key_version: "encryptionkey@0"
I20260619 02:24:46.532302 2202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 02:24:46.534657 2217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:46.535812 2202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:24:46.535943 2202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "02258e2a40a346ea80523911c8542b20"
format_stamp: "Formatted at 2026-06-19 02:24:46 on dist-test-slave-xkrt"
server_key: "db42e45e1a3e589209e5ff56ba81cdbf"
server_key_iv: "4ee7185de96001c479475ab094df58b6"
server_key_version: "encryptionkey@0"
I20260619 02:24:46.536047 2202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:46.549049 2202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:46.552718 2202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:46.552913 2202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:46.554423 2202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:46.554499 2202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:46.554569 2202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:46.554597 2202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:24:46 dist-test-slave-xkrt krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:46.565032 2202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33287
I20260619 02:24:46.565117 2332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33287 every 8 connection(s)
I20260619 02:24:46.565986 2202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:24:46.569939 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:46.555253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50625 (local address 127.0.98.62:43821)
0619 02:24:46.555373 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:46.555394 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:46.556442 (+ 1048us) server_negotiation.cc:407] Connection header received
0619 02:24:46.557586 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:46.557592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:46.557663 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:46.557766 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:46.559281 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.560172 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.562104 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.562331 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.562423 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:46.565470 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:46.565474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:46.565477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:46.565509 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:46.567933 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:46.568438 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:46.568443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:46.568445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:46.568531 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:46.568859 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:46.568864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:46.568866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:46.569050 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:46.569156 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:46.569647 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 02:24:46.569761 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:24:46.570621 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:46.555808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43821 (local address 127.0.98.1:50625)
0619 02:24:46.556289 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:46.556328 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:46.557295 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:46.557941 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:46.557951 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:46.558433 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:46.559106 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:46.559129 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.560343 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.560348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:46.561984 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:46.561998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.562506 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.562511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:46.562720 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:46.563401 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:46.563423 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:46.565292 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:46.568060 (+ 2768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:46.568068 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:46.568083 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:46.568329 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:46.568623 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:46.568627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:46.568629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:46.568763 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:46.569153 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:46.569160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:46.569393 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 02:24:46.569585 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 02:24:46.569878 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":141,"threads_started":1}
I20260619 02:24:46.571882 2333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43821
I20260619 02:24:46.572042 2333 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:46.572507 2333 heartbeater.cc:511] Master 127.0.98.62:43821 requested a full tablet report, sending...
I20260619 02:24:46.574184 2130 ts_manager.cc:194] Registered new tserver with Master: 02258e2a40a346ea80523911c8542b20 (127.0.98.1:33287)
I20260619 02:24:46.574930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 2202
I20260619 02:24:46.575047 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:24:46.575347 392 external_mini_cluster.cc:1428] Setting key f168ce74301472b823cfd57c90abe795
I20260619 02:24:46.575692 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50625
I20260619 02:24:46.583353 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:46.574148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51483 (local address 127.0.98.62:43821)
0619 02:24:46.576330 (+ 2182us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:46.576334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:46.576346 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:24:46.576379 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:46.576383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:46.576429 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:46.576506 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:46.577005 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.577873 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.578724 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.579127 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.579245 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:46.580252 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:46.580255 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:46.580258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:46.580281 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:46.581775 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:46.582277 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:46.582280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:46.582281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:46.582350 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:46.582654 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:46.582656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:46.582658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:46.582799 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:46.582876 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:46.583154 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 02:24:46.583245 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2113}
I20260619 02:24:46.583516 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:46.574029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43821 (local address 127.0.98.1:51483)
0619 02:24:46.574312 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:46.574327 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:46.574421 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:46.576526 (+ 2105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:46.576529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:46.576740 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:46.576874 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:46.576881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.578010 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.578013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:46.578604 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:46.578614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.579242 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.579245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:46.579309 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:46.579804 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:46.579814 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:46.580119 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:46.581885 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:46.581891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:46.581895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:46.582178 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:46.582452 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:46.582456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:46.582458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:46.582558 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:46.582895 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:46.582899 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:46.582995 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 02:24:46.583107 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:24:46.583276 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:24:46.601713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43821
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:46.709230 2342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:46.709455 2342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:46.709498 2342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:46.713418 2342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:46.713469 2342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:46.713506 2342 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:24:46.713550 2342 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:24:46.713634 2342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:24:46.718672 2342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:46.719581 2342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:46.720502 2342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:46.729593 2347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:46.729619 2348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:46.729812 2342 server_base.cc:1034] running on GCE node
W20260619 02:24:46.729835 2350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:46.730180 2342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:46.730795 2342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:46.731972 2342 hybrid_clock.cc:648] HybridClock initialized: now 1781835886731958 us; error 26 us; skew 500 ppm
Jun 19 02:24:46 dist-test-slave-xkrt krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:46.734604 2342 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:24:46.735396 2342 webserver.cc:460] Webserver started at http://127.0.98.2:39421/ using document root <none> and password file <none>
I20260619 02:24:46.735875 2342 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:46.735930 2342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:46.736073 2342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:46.737982 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "b9cbec7e31164b49b493d60dc7258845"
format_stamp: "Formatted at 2026-06-19 02:24:46 on dist-test-slave-xkrt"
server_key: "4cfa79d7f38d6f24bf68343ed623c838"
server_key_iv: "96cccbcccdef89695e8dab07e0024a6b"
server_key_version: "encryptionkey@0"
I20260619 02:24:46.738458 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b9cbec7e31164b49b493d60dc7258845"
format_stamp: "Formatted at 2026-06-19 02:24:46 on dist-test-slave-xkrt"
server_key: "4cfa79d7f38d6f24bf68343ed623c838"
server_key_iv: "96cccbcccdef89695e8dab07e0024a6b"
server_key_version: "encryptionkey@0"
I20260619 02:24:46.741981 2342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:24:46.744149 2357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:46.745254 2342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:24:46.745340 2342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "b9cbec7e31164b49b493d60dc7258845"
format_stamp: "Formatted at 2026-06-19 02:24:46 on dist-test-slave-xkrt"
server_key: "4cfa79d7f38d6f24bf68343ed623c838"
server_key_iv: "96cccbcccdef89695e8dab07e0024a6b"
server_key_version: "encryptionkey@0"
I20260619 02:24:46.745412 2342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:46.754769 2342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:46.757264 2342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:46.757400 2342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:46.758750 2342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:46.758798 2342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:46.758836 2342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:46.758888 2342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:24:46 dist-test-slave-xkrt krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:46.769318 2342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38529
I20260619 02:24:46.769364 2472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38529 every 8 connection(s)
I20260619 02:24:46.770308 2342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:24:46.772393 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:46.759428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57601 (local address 127.0.98.62:43821)
0619 02:24:46.759549 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:46.759553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:46.760535 (+ 982us) server_negotiation.cc:407] Connection header received
0619 02:24:46.761491 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:46.761495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:46.761572 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:46.761686 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:46.762985 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.763592 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.764861 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.765085 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.765196 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:46.768737 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:46.768740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:46.768743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:46.768766 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:46.770304 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:46.770834 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:46.770837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:46.770839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:46.770924 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:46.771298 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:46.771301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:46.771302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:46.771477 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:46.771554 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:46.772098 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 02:24:46.772223 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:24:46.773159 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:46.759944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43821 (local address 127.0.98.2:57601)
0619 02:24:46.760385 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:46.760435 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:46.761283 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:46.761977 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:46.761983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:46.762393 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:46.762836 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:46.762848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.763715 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.763719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:46.764742 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:46.764751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.765234 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.765237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:46.765448 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:46.766154 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:46.766187 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:46.768570 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:46.770430 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:46.770435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:46.770448 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:46.770691 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:46.771032 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:46.771037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:46.771039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:46.771194 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:46.771587 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:46.771594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:46.771841 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 02:24:46.772008 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 02:24:46.772320 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":131,"threads_started":1}
I20260619 02:24:46.774283 2473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43821
I20260619 02:24:46.774439 2473 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:46.774881 2473 heartbeater.cc:511] Master 127.0.98.62:43821 requested a full tablet report, sending...
I20260619 02:24:46.775847 2129 ts_manager.cc:194] Registered new tserver with Master: b9cbec7e31164b49b493d60dc7258845 (127.0.98.2:38529)
I20260619 02:24:46.776443 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57601
I20260619 02:24:46.777011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 2342
I20260619 02:24:46.777097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:24:46.777375 392 external_mini_cluster.cc:1428] Setting key 66d053fdd9a7450e95421e14fc09e212
I20260619 02:24:46.784864 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:46.776381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43821 (local address 127.0.98.2:33941)
0619 02:24:46.776648 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:46.776663 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:46.776758 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:46.778119 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:46.778122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:46.778325 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:46.778435 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:46.778443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.779176 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.779182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:46.780184 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:46.780198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.780674 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.780677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:46.780765 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:46.781300 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:46.781311 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:46.781694 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:46.783358 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:46.783361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:46.783363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:46.783561 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:46.783836 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:46.783839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:46.783841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:46.783931 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:46.784341 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:46.784347 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:46.784437 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 02:24:46.784544 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 02:24:46.784682 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
I20260619 02:24:46.784858 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:46.776432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33941 (local address 127.0.98.62:43821)
0619 02:24:46.776557 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:46.776561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:46.776757 (+ 196us) server_negotiation.cc:407] Connection header received
0619 02:24:46.777872 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:46.777875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:46.777927 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:46.778015 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:46.778551 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.779050 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.780343 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.780560 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.780671 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:46.781821 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:46.781829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:46.781832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:46.781855 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:46.783229 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:46.783660 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:46.783664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:46.783666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:46.783740 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:46.784021 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:46.784025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:46.784027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:46.784226 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:46.784331 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:46.784597 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 02:24:46.784702 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:24:46.804456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43821
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:46.912118 2482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:46.912358 2482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:46.912403 2482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:46.916262 2482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:46.916313 2482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:46.916350 2482 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:24:46.916364 2482 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:24:46.916450 2482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:24:46.921746 2482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:46.922712 2482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:46.923664 2482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:46.932780 2487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:46.932780 2490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:46.932914 2482 server_base.cc:1034] running on GCE node
W20260619 02:24:46.933073 2488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:46.933449 2482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:46.934078 2482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:46.935236 2482 hybrid_clock.cc:648] HybridClock initialized: now 1781835886935219 us; error 30 us; skew 500 ppm
Jun 19 02:24:46 dist-test-slave-xkrt krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:46.937774 2482 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:24:46.938845 2482 webserver.cc:460] Webserver started at http://127.0.98.3:46559/ using document root <none> and password file <none>
I20260619 02:24:46.939478 2482 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:46.939568 2482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:46.939788 2482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:46.941850 2482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "8c44b84db3f34bf294eb7755eb8fb61b"
format_stamp: "Formatted at 2026-06-19 02:24:46 on dist-test-slave-xkrt"
server_key: "6f6b786cb721ca1b25dae6b3687a2724"
server_key_iv: "411d757d6532dfc519eb699611b6ace1"
server_key_version: "encryptionkey@0"
I20260619 02:24:46.942394 2482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8c44b84db3f34bf294eb7755eb8fb61b"
format_stamp: "Formatted at 2026-06-19 02:24:46 on dist-test-slave-xkrt"
server_key: "6f6b786cb721ca1b25dae6b3687a2724"
server_key_iv: "411d757d6532dfc519eb699611b6ace1"
server_key_version: "encryptionkey@0"
I20260619 02:24:46.945995 2482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 02:24:46.948318 2497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:46.949594 2482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:24:46.949736 2482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "8c44b84db3f34bf294eb7755eb8fb61b"
format_stamp: "Formatted at 2026-06-19 02:24:46 on dist-test-slave-xkrt"
server_key: "6f6b786cb721ca1b25dae6b3687a2724"
server_key_iv: "411d757d6532dfc519eb699611b6ace1"
server_key_version: "encryptionkey@0"
I20260619 02:24:46.949841 2482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:46.959087 2482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:46.961742 2482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:46.961917 2482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:46.963196 2482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:46.963276 2482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:46.963344 2482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:46.963395 2482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:24:46 dist-test-slave-xkrt krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:46.974123 2482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39265
I20260619 02:24:46.974187 2612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39265 every 8 connection(s)
I20260619 02:24:46.975196 2482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:24:46.976624 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:46.964015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51099 (local address 127.0.98.62:43821)
0619 02:24:46.964170 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:46.964176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:46.964950 (+ 774us) server_negotiation.cc:407] Connection header received
0619 02:24:46.965978 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:46.965981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:46.966038 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:46.966119 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:46.967709 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.968575 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.969870 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.970173 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.970288 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:46.972788 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:46.972794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:46.972800 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:46.972834 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:46.974546 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:46.975028 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:46.975032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:46.975034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:46.975123 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:46.975521 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:46.975525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:46.975526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:46.975747 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:46.975853 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:46.976412 (+ 559us) server_negotiation.cc:299] Negotiation successful
0619 02:24:46.976507 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:24:46.977444 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:46.964370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43821 (local address 127.0.98.3:51099)
0619 02:24:46.964797 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:46.964843 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:46.965774 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:46.966392 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:46.966401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:46.966891 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:46.967554 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:46.967576 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.968694 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.968698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:46.969762 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:46.969774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.970328 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.970332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:46.970512 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:46.971054 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:46.971076 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:46.972607 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:46.974645 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:46.974650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:46.974663 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:46.974926 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:46.975218 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:46.975220 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:46.975221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:46.975413 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:46.975907 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:46.975913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:46.976176 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 02:24:46.976349 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 02:24:46.976571 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":119,"threads_started":1}
I20260619 02:24:46.978509 2613 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43821
I20260619 02:24:46.978657 2613 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:46.979121 2613 heartbeater.cc:511] Master 127.0.98.62:43821 requested a full tablet report, sending...
I20260619 02:24:46.980096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 2482
I20260619 02:24:46.980123 2130 ts_manager.cc:194] Registered new tserver with Master: 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265)
I20260619 02:24:46.980221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:24:46.980546 392 external_mini_cluster.cc:1428] Setting key 454152469d0be0310ff0cc9942500d0e
I20260619 02:24:46.980820 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51099
I20260619 02:24:46.982676 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:24:46.987944 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:46.980442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43821 (local address 127.0.98.3:47569)
0619 02:24:46.980708 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:46.980719 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:46.980805 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:46.981277 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:46.981279 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:46.981474 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:46.981692 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:46.981701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.982639 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.982641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:46.983267 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:46.983275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.983722 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.983725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:46.983808 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:46.984328 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:46.984364 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:46.984688 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:46.986414 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:46.986420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:46.986425 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:46.986695 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:46.987014 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:46.987019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:46.987023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:46.987118 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:46.987456 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:46.987459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:46.987554 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 02:24:46.987668 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 02:24:46.987807 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260619 02:24:46.988029 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:46.980887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47569 (local address 127.0.98.62:43821)
0619 02:24:46.981101 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:46.981105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:46.981118 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:24:46.981147 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:46.981149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:46.981184 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:46.981266 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:46.981812 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.982518 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.983381 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:46.983609 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:46.983716 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:46.984821 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:46.984827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:46.984831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:46.984859 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:46.986301 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:46.986824 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:46.986828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:46.986830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:46.986917 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:46.987211 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:46.987213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:46.987215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:46.987362 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:46.987455 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:46.987760 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 02:24:46.987872 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
Jun 19 02:24:46 dist-test-slave-xkrt krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:24:46 dist-test-slave-xkrt krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:47.007794 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.000159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.0.98.62:43821)
0619 02:24:47.000289 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:47.000293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:47.000303 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:24:47.000336 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:47.000338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:47.000371 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:47.000462 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:47.001122 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.001693 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.002545 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.002763 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.002874 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.005091 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:47.005093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:47.005095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:47.005116 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:47.006523 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:47.006876 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:47.006879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:47.006880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:47.006946 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:47.007159 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:47.007161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:47.007162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:47.007309 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:47.007405 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:47.007596 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 02:24:47.007683 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:24:47.010412 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:24:47.013269 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:24:47.026953 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.021035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33287 (local address 127.0.0.1:40090)
0619 02:24:47.022104 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:47.022130 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:47.022236 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:47.022767 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:47.022790 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:47.022818 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:47.022942 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.022947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.024095 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.024101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.025442 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.025450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.026420 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.026426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.026542 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.026584 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 02:24:47.026691 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 02:24:47.026799 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":95,"threads_started":1}
I20260619 02:24:47.026983 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.021767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.0.98.1:33287)
0619 02:24:47.022163 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:47.022176 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:47.022201 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:24:47.022427 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:47.022433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:47.022625 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:47.022768 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:47.023062 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.024003 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.025578 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.026296 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.026407 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.026556 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:47.026732 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 02:24:47.026846 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":112,"threads_started":1}
I20260619 02:24:47.027500 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.020958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38529 (local address 127.0.0.1:59438)
0619 02:24:47.021389 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:47.021432 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:47.021574 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:47.023359 (+ 1785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:47.023362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:47.023379 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:47.023475 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.023482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.024700 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.024704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.025867 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.025877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.027083 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.027087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.027178 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.027200 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:24:47.027295 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 02:24:47.027377 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260619 02:24:47.027546 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.021082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.0.98.2:38529)
0619 02:24:47.023006 (+ 1924us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:47.023014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:47.023031 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:24:47.023085 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:47.023088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:47.023228 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:47.023377 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:47.023607 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.024581 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.026005 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.026784 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.026880 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.027015 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:47.027337 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 02:24:47.027418 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1844,"thread_start_us":92,"threads_started":1}
I20260619 02:24:47.028153 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.021830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39265 (local address 127.0.0.1:48008)
0619 02:24:47.022354 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:47.022367 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:47.022461 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:47.023952 (+ 1491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:47.023955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:47.023972 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:47.024050 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.024059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.025390 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.025393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.026333 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.026340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.027587 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.027780 (+ 193us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.027866 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.027888 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:24:47.027960 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 02:24:47.028025 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":85,"threads_started":1}
I20260619 02:24:47.028578 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.021995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.0.98.3:39265)
0619 02:24:47.023634 (+ 1639us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:47.023642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:47.023658 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:24:47.023708 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:47.023712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:47.023826 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:47.023961 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:47.024172 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.025274 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.026493 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.027251 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.027358 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.027527 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:47.028353 (+ 826us) server_negotiation.cc:299] Negotiation successful
0619 02:24:47.028451 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1545,"thread_start_us":108,"threads_started":1}
I20260619 02:24:47.030211 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0f2a86fa7144f6b776228dd371f553 (DEFAULT_TABLE table=test-table [id=58ad9352b118430eb80e5156d8c4ab6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:47.030292 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0f2a86fa7144f6b776228dd371f553 (DEFAULT_TABLE table=test-table [id=58ad9352b118430eb80e5156d8c4ab6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:47.031167 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e0f2a86fa7144f6b776228dd371f553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:47.030936 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0f2a86fa7144f6b776228dd371f553 (DEFAULT_TABLE table=test-table [id=58ad9352b118430eb80e5156d8c4ab6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:47.031275 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e0f2a86fa7144f6b776228dd371f553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:47.031816 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e0f2a86fa7144f6b776228dd371f553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:47.037772 2636 tablet_bootstrap.cc:492] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845: Bootstrap starting.
I20260619 02:24:47.039239 2637 tablet_bootstrap.cc:492] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20: Bootstrap starting.
I20260619 02:24:47.040082 2636 tablet_bootstrap.cc:654] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:47.040807 2636 log.cc:824] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:47.041057 2637 tablet_bootstrap.cc:654] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:47.041473 2638 tablet_bootstrap.cc:492] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b: Bootstrap starting.
I20260619 02:24:47.041813 2637 log.cc:824] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:47.042898 2636 tablet_bootstrap.cc:492] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845: No bootstrap required, opened a new log
I20260619 02:24:47.043107 2636 ts_tablet_manager.cc:1397] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:24:47.043704 2637 tablet_bootstrap.cc:492] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20: No bootstrap required, opened a new log
I20260619 02:24:47.043715 2638 tablet_bootstrap.cc:654] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:47.043901 2637 ts_tablet_manager.cc:1397] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 02:24:47.044471 2638 log.cc:824] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:47.046324 2638 tablet_bootstrap.cc:492] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b: No bootstrap required, opened a new log
I20260619 02:24:47.046502 2638 ts_tablet_manager.cc:1397] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:24:47.046422 2636 raft_consensus.cc:348] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.046712 2636 raft_consensus.cc:374] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:47.046765 2636 raft_consensus.cc:729] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9cbec7e31164b49b493d60dc7258845, State: Initialized, Role: FOLLOWER
I20260619 02:24:47.047209 2636 consensus_queue.cc:260] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.047574 2637 raft_consensus.cc:348] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.047821 2637 raft_consensus.cc:374] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:47.047855 2473 heartbeater.cc:503] Master 127.0.98.62:43821 was elected leader, sending a full tablet report...
I20260619 02:24:47.047883 2637 raft_consensus.cc:729] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02258e2a40a346ea80523911c8542b20, State: Initialized, Role: FOLLOWER
I20260619 02:24:47.048271 2637 consensus_queue.cc:260] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.049219 2637 ts_tablet_manager.cc:1428] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:24:47.049232 2636 ts_tablet_manager.cc:1428] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:24:47.049573 2333 heartbeater.cc:503] Master 127.0.98.62:43821 was elected leader, sending a full tablet report...
I20260619 02:24:47.052120 2638 raft_consensus.cc:348] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.052388 2638 raft_consensus.cc:374] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:47.052457 2638 raft_consensus.cc:729] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c44b84db3f34bf294eb7755eb8fb61b, State: Initialized, Role: FOLLOWER
I20260619 02:24:47.052819 2638 consensus_queue.cc:260] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.053452 2613 heartbeater.cc:503] Master 127.0.98.62:43821 was elected leader, sending a full tablet report...
I20260619 02:24:47.053673 2638 ts_tablet_manager.cc:1428] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260619 02:24:47.067328 2334 tablet.cc:2369] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:24:47.226225 2614 tablet.cc:2369] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:24:47.271934 2474 tablet.cc:2369] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:24:47.304085 2642 raft_consensus.cc:482] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:24:47.304266 2642 raft_consensus.cc:504] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.305500 2642 leader_election.cc:288] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02258e2a40a346ea80523911c8542b20 (127.0.98.1:33287), 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265)
I20260619 02:24:47.310619 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.305795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54101 (local address 127.0.98.1:33287)
0619 02:24:47.305924 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:47.305930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:47.306394 (+ 464us) server_negotiation.cc:407] Connection header received
0619 02:24:47.306541 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:47.306545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:47.306598 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:47.306683 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:47.307008 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.307666 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.309466 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.309982 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.310071 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.310149 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:47.310389 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 02:24:47.310493 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:24:47.310636 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.305905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34115 (local address 127.0.98.3:39265)
0619 02:24:47.306012 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:47.306016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:47.306248 (+ 232us) server_negotiation.cc:407] Connection header received
0619 02:24:47.306447 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:47.306450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:47.306498 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:47.306547 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:47.306992 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.307781 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.309634 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.310064 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.310170 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.310236 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:47.310464 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 02:24:47.310535 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:24:47.310629 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.306017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33287 (local address 127.0.98.2:54101)
0619 02:24:47.306299 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:47.306311 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:47.306431 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:47.306693 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:47.306696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:47.306710 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:47.306876 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.306883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.307802 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.307805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.309325 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.309336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.310111 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.310117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.310253 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.310279 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 02:24:47.310368 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 02:24:47.310471 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":135,"thread_start_us":82,"threads_started":1}
I20260619 02:24:47.310645 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.305786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39265 (local address 127.0.98.2:34115)
0619 02:24:47.306162 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:47.306176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:47.306328 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:47.306684 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:47.306687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:47.306701 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:47.306860 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.306866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.307868 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.307871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.309325 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.309342 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.310203 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.310206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.310306 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.310342 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:24:47.310431 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 02:24:47.310501 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":183,"thread_start_us":70,"threads_started":1}
I20260619 02:24:47.311268 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0f2a86fa7144f6b776228dd371f553" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02258e2a40a346ea80523911c8542b20" is_pre_election: true
I20260619 02:24:47.311352 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0f2a86fa7144f6b776228dd371f553" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" is_pre_election: true
I20260619 02:24:47.311582 2567 raft_consensus.cc:2454] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 0.
I20260619 02:24:47.311583 2287 raft_consensus.cc:2454] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 0.
I20260619 02:24:47.312062 2358 leader_election.cc:302] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c44b84db3f34bf294eb7755eb8fb61b, b9cbec7e31164b49b493d60dc7258845; no voters:
I20260619 02:24:47.312340 2642 raft_consensus.cc:2781] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:24:47.312419 2642 raft_consensus.cc:482] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:24:47.312449 2642 raft_consensus.cc:3037] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:47.313452 2642 raft_consensus.cc:504] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.313855 2642 leader_election.cc:288] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 election: Requested vote from peers 02258e2a40a346ea80523911c8542b20 (127.0.98.1:33287), 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265)
I20260619 02:24:47.314128 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0f2a86fa7144f6b776228dd371f553" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02258e2a40a346ea80523911c8542b20"
I20260619 02:24:47.314248 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0f2a86fa7144f6b776228dd371f553" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c44b84db3f34bf294eb7755eb8fb61b"
I20260619 02:24:47.314270 2287 raft_consensus.cc:3037] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:47.314392 2567 raft_consensus.cc:3037] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:47.315275 2287 raft_consensus.cc:2454] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 1.
I20260619 02:24:47.315568 2361 leader_election.cc:302] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02258e2a40a346ea80523911c8542b20, b9cbec7e31164b49b493d60dc7258845; no voters:
I20260619 02:24:47.315714 2642 raft_consensus.cc:2781] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:47.315922 2642 raft_consensus.cc:686] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [term 1 LEADER]: Becoming Leader. State: Replica: b9cbec7e31164b49b493d60dc7258845, State: Running, Role: LEADER
I20260619 02:24:47.315965 2567 raft_consensus.cc:2454] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 1.
I20260619 02:24:47.316241 2642 consensus_queue.cc:237] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.319167 2129 catalog_manager.cc:5495] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9cbec7e31164b49b493d60dc7258845 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b9cbec7e31164b49b493d60dc7258845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:24:47 dist-test-slave-xkrt krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:47.407434 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.399423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.0.98.62:43821)
0619 02:24:47.399582 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:47.399586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:47.399687 (+ 101us) server_negotiation.cc:407] Connection header received
0619 02:24:47.399797 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:47.399800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:47.399839 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:47.399917 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:47.400572 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.401089 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.401892 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.402107 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.402183 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.404405 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:47.404408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:47.404410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:47.404430 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:47.406001 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:47.406406 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:47.406409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:47.406410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:47.406482 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:47.406716 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:47.406719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:47.406720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:47.406871 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:47.406950 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:47.407217 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 02:24:47.407300 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 02:24:47.407437 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.399333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43821 (local address 127.0.0.1:46352)
0619 02:24:47.399600 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:47.399613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:47.399718 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:47.399944 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:47.399946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:47.400373 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:47.400465 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.400471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.401218 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.401220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.401800 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.401807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.402228 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.402231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.402293 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.402823 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:47.402835 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:47.404259 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:47.406089 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:47.406097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:47.406102 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:47.406315 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:47.406560 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:47.406562 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:47.406564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:47.406636 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:47.406968 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:47.406972 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:47.407065 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 02:24:47.407151 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:24:47.407250 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
I20260619 02:24:47.411219 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46352:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:24:47.412001 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:24:47.417227 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 7fcd2bc4d1ea4fa4b8cffaabe4185c39 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbcc03283ada416db695c406d300dd2b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:24:47.417462 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 7fcd2bc4d1ea4fa4b8cffaabe4185c39 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbcc03283ada416db695c406d300dd2b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:24:47.417573 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 7fcd2bc4d1ea4fa4b8cffaabe4185c39 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbcc03283ada416db695c406d300dd2b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:24:47.418047 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fcd2bc4d1ea4fa4b8cffaabe4185c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:47.418169 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fcd2bc4d1ea4fa4b8cffaabe4185c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:47.418196 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fcd2bc4d1ea4fa4b8cffaabe4185c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:47.420995 2636 tablet_bootstrap.cc:492] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845: Bootstrap starting.
I20260619 02:24:47.421022 2637 tablet_bootstrap.cc:492] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20: Bootstrap starting.
I20260619 02:24:47.421952 2637 tablet_bootstrap.cc:654] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:47.422362 2636 tablet_bootstrap.cc:654] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:47.423257 2637 tablet_bootstrap.cc:492] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20: No bootstrap required, opened a new log
I20260619 02:24:47.423362 2637 ts_tablet_manager.cc:1397] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:24:47.423610 2636 tablet_bootstrap.cc:492] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845: No bootstrap required, opened a new log
I20260619 02:24:47.423678 2636 ts_tablet_manager.cc:1397] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:24:47.423869 2637 raft_consensus.cc:348] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.423980 2637 raft_consensus.cc:374] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:47.424023 2637 raft_consensus.cc:729] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02258e2a40a346ea80523911c8542b20, State: Initialized, Role: FOLLOWER
I20260619 02:24:47.424113 2636 raft_consensus.cc:348] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.424149 2637 consensus_queue.cc:260] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.424226 2636 raft_consensus.cc:374] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:47.424252 2636 raft_consensus.cc:729] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9cbec7e31164b49b493d60dc7258845, State: Initialized, Role: FOLLOWER
I20260619 02:24:47.424396 2636 consensus_queue.cc:260] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.424422 2637 ts_tablet_manager.cc:1428] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:24:47.424641 2636 ts_tablet_manager.cc:1428] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:24:47.424616 2642 tablet_replica.cc:439] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } } }
I20260619 02:24:47.424736 2642 tablet_replica.cc:442] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:24:47.424943 2638 tablet_bootstrap.cc:492] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b: Bootstrap starting.
I20260619 02:24:47.425778 2643 tablet_replica.cc:439] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } } }
I20260619 02:24:47.425855 2638 tablet_bootstrap.cc:654] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:47.425930 2643 tablet_replica.cc:442] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:24:47.427404 2638 tablet_bootstrap.cc:492] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b: No bootstrap required, opened a new log
I20260619 02:24:47.427497 2638 ts_tablet_manager.cc:1397] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:24:47.427874 2638 raft_consensus.cc:348] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.427963 2638 raft_consensus.cc:374] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:47.428015 2638 raft_consensus.cc:729] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c44b84db3f34bf294eb7755eb8fb61b, State: Initialized, Role: FOLLOWER
I20260619 02:24:47.428135 2638 consensus_queue.cc:260] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.428383 2638 ts_tablet_manager.cc:1428] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:24:47.428406 2644 tablet_replica.cc:439] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } } }
I20260619 02:24:47.428563 2644 tablet_replica.cc:442] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:24:47.485881 2642 raft_consensus.cc:482] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:24:47.486016 2642 raft_consensus.cc:504] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.486419 2642 leader_election.cc:288] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02258e2a40a346ea80523911c8542b20 (127.0.98.1:33287), 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265)
I20260619 02:24:47.486840 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcd2bc4d1ea4fa4b8cffaabe4185c39" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02258e2a40a346ea80523911c8542b20" is_pre_election: true
I20260619 02:24:47.486867 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcd2bc4d1ea4fa4b8cffaabe4185c39" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" is_pre_election: true
I20260619 02:24:47.486997 2287 raft_consensus.cc:2454] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 0.
I20260619 02:24:47.487090 2567 raft_consensus.cc:2454] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 0.
I20260619 02:24:47.487308 2361 leader_election.cc:302] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02258e2a40a346ea80523911c8542b20, b9cbec7e31164b49b493d60dc7258845; no voters:
I20260619 02:24:47.487479 2642 raft_consensus.cc:2781] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:24:47.487537 2642 raft_consensus.cc:482] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:24:47.487586 2642 raft_consensus.cc:3037] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:47.488607 2642 raft_consensus.cc:504] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.488982 2642 leader_election.cc:288] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 election: Requested vote from peers 02258e2a40a346ea80523911c8542b20 (127.0.98.1:33287), 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265)
I20260619 02:24:47.489300 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcd2bc4d1ea4fa4b8cffaabe4185c39" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02258e2a40a346ea80523911c8542b20"
I20260619 02:24:47.489375 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcd2bc4d1ea4fa4b8cffaabe4185c39" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c44b84db3f34bf294eb7755eb8fb61b"
I20260619 02:24:47.489439 2287 raft_consensus.cc:3037] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:47.489501 2567 raft_consensus.cc:3037] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:47.490368 2287 raft_consensus.cc:2454] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 1.
I20260619 02:24:47.490417 2567 raft_consensus.cc:2454] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 1.
I20260619 02:24:47.490629 2361 leader_election.cc:302] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02258e2a40a346ea80523911c8542b20, b9cbec7e31164b49b493d60dc7258845; no voters:
I20260619 02:24:47.490797 2642 raft_consensus.cc:2781] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:47.490914 2642 raft_consensus.cc:686] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [term 1 LEADER]: Becoming Leader. State: Replica: b9cbec7e31164b49b493d60dc7258845, State: Running, Role: LEADER
I20260619 02:24:47.491048 2642 consensus_queue.cc:237] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.491775 2647 tablet_replica.cc:439] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845: TxnStatusTablet state changed. Reason: New leader b9cbec7e31164b49b493d60dc7258845. Latest consensus state: current_term: 1 leader_uuid: "b9cbec7e31164b49b493d60dc7258845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } } }
I20260619 02:24:47.491917 2647 tablet_replica.cc:442] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:24:47.492319 2659 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:24:47.492551 2659 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:24:47.492827 2129 catalog_manager.cc:5495] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9cbec7e31164b49b493d60dc7258845 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b9cbec7e31164b49b493d60dc7258845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } health_report { overall_health: UNKNOWN } } }
I20260619 02:24:47.523548 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.519445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38529 (local address 127.0.0.1:59452)
0619 02:24:47.519571 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:47.519588 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:47.519691 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:47.520037 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:47.520041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:47.520050 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:24:47.520157 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.520162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.520999 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.521005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.521770 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.521778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.522363 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.522365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.522426 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.522475 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:24:47.523292 (+ 817us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:24:47.523305 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 02:24:47.523361 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:24:47.523438 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 02:24:47.523577 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.519543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.0.98.2:38529)
0619 02:24:47.519675 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:47.519680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:47.519693 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:24:47.519851 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:47.519859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:47.519924 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:47.520027 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:24:47.520276 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.520892 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.521909 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.522257 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.522364 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.522629 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:24:47.523187 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:24:47.523293 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:47.523375 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 02:24:47.523442 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:24:47.528685 2287 raft_consensus.cc:1261] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Refusing update from remote peer b9cbec7e31164b49b493d60dc7258845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:47.528694 2567 raft_consensus.cc:1261] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Refusing update from remote peer b9cbec7e31164b49b493d60dc7258845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:47.529371 2642 consensus_queue.cc:1035] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:24:47.529626 2647 consensus_queue.cc:1035] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:24:47.533257 2644 tablet_replica.cc:439] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b: TxnStatusTablet state changed. Reason: New leader b9cbec7e31164b49b493d60dc7258845. Latest consensus state: current_term: 1 leader_uuid: "b9cbec7e31164b49b493d60dc7258845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } } }
I20260619 02:24:47.533375 2644 tablet_replica.cc:442] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:24:47.533926 2643 tablet_replica.cc:439] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20: TxnStatusTablet state changed. Reason: New leader b9cbec7e31164b49b493d60dc7258845. Latest consensus state: current_term: 1 leader_uuid: "b9cbec7e31164b49b493d60dc7258845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } } }
I20260619 02:24:47.534039 2643 tablet_replica.cc:442] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:24:47.538009 2644 tablet_replica.cc:439] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9cbec7e31164b49b493d60dc7258845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } } }
I20260619 02:24:47.538115 2644 tablet_replica.cc:442] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 8c44b84db3f34bf294eb7755eb8fb61b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:24:47.538009 2667 tablet_replica.cc:439] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9cbec7e31164b49b493d60dc7258845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } } }
I20260619 02:24:47.538158 2667 tablet_replica.cc:442] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:24:47.538277 2642 tablet_replica.cc:439] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9cbec7e31164b49b493d60dc7258845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } } }
I20260619 02:24:47.538352 2642 tablet_replica.cc:442] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P b9cbec7e31164b49b493d60dc7258845: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:24:47.539141 2643 tablet_replica.cc:439] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9cbec7e31164b49b493d60dc7258845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } } }
I20260619 02:24:47.539259 2643 tablet_replica.cc:442] T 7fcd2bc4d1ea4fa4b8cffaabe4185c39 P 02258e2a40a346ea80523911c8542b20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:24:47.543143 2665 mvcc.cc:205] Tried to move back new op lower bound from 7298399795310665728 to 7298399795164684288. Current Snapshot: MvccSnapshot[applied={T|T < 7298399795310665728}]
I20260619 02:24:47.544155 2663 mvcc.cc:205] Tried to move back new op lower bound from 7298399795310665728 to 7298399795164684288. Current Snapshot: MvccSnapshot[applied={T|T < 7298399795310665728 or (T in {7298399795310665728})}]
I20260619 02:24:47.547588 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.544184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.0.98.2:38529)
0619 02:24:47.544312 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:47.544316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:47.544587 (+ 271us) server_negotiation.cc:407] Connection header received
0619 02:24:47.544735 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:47.544738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:47.544781 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:47.544869 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:24:47.545110 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.545745 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.546617 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.546831 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.546936 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.547184 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:24:47.547256 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:24:47.547334 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:47.547419 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 02:24:47.547462 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:24:47.558171 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.554725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47895 (local address 127.0.98.2:38529)
0619 02:24:47.554852 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:47.554857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:47.555057 (+ 200us) server_negotiation.cc:407] Connection header received
0619 02:24:47.555182 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:47.555184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:47.555220 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:47.555271 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:24:47.555541 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.555994 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.557188 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.557376 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.557447 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.557801 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:24:47.557865 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:24:47.557920 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:47.558025 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 02:24:47.558075 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":48}
I20260619 02:24:47.558306 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:47.554676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38529 (local address 127.0.98.2:47895)
0619 02:24:47.554974 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:47.554988 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:47.555087 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:47.555300 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:47.555303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:47.555310 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:24:47.555445 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.555451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.556088 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.556091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.557085 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:47.557095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:47.557493 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:47.557496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:47.557617 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:47.557662 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:24:47.557945 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:24:47.557951 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:24:47.558020 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 02:24:47.558121 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":17,"thread_start_us":113,"threads_started":1}
I20260619 02:24:47.564260 2567 raft_consensus.cc:1261] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Refusing update from remote peer b9cbec7e31164b49b493d60dc7258845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:47.564409 2287 raft_consensus.cc:1261] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Refusing update from remote peer b9cbec7e31164b49b493d60dc7258845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:47.564659 2670 consensus_queue.cc:1035] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:24:47.564957 2670 consensus_queue.cc:1035] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:24:47.568379 2662 mvcc.cc:205] Tried to move back new op lower bound from 7298399795460132864 to 7298399794449559552. Current Snapshot: MvccSnapshot[applied={T|T < 7298399795460132864}]
I20260619 02:24:47.611544 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:24:47.611779 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:24:47.614051 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2: Sending DeleteTablet for 3 replicas of tablet 6e0f2a86fa7144f6b776228dd371f553
I20260619 02:24:47.614722 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0f2a86fa7144f6b776228dd371f553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48008
I20260619 02:24:47.614820 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0f2a86fa7144f6b776228dd371f553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40090
I20260619 02:24:47.615334 2681 tablet_replica.cc:330] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20: stopping tablet replica
I20260619 02:24:47.615353 2680 tablet_replica.cc:330] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b: stopping tablet replica
I20260619 02:24:47.615425 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0f2a86fa7144f6b776228dd371f553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59438
I20260619 02:24:47.615643 2681 raft_consensus.cc:2229] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:24:47.615731 2680 raft_consensus.cc:2229] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:24:47.615899 2681 raft_consensus.cc:2258] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:47.615845 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 02:24:47.616178 2680 raft_consensus.cc:2258] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 02:24:47.616220 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:24:47.617236 2681 ts_tablet_manager.cc:1905] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:47.617261 2680 ts_tablet_manager.cc:1905] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:47.617762 2682 tablet_replica.cc:330] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845: stopping tablet replica
I20260619 02:24:47.618026 2682 raft_consensus.cc:2229] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:24:47.618331 2682 raft_consensus.cc:2258] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:47.619391 2682 ts_tablet_manager.cc:1905] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:47.620338 2681 ts_tablet_manager.cc:1918] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:24:47.620433 2681 log.cc:1196] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/wal/wals/6e0f2a86fa7144f6b776228dd371f553
I20260619 02:24:47.620738 2681 ts_tablet_manager.cc:1939] T 6e0f2a86fa7144f6b776228dd371f553 P 02258e2a40a346ea80523911c8542b20: Deleting consensus metadata
I20260619 02:24:47.622156 2407 tablet_service.cc:1467] Processing CreateTablet for tablet e4b16f5e3f73443eb4c67f4eb81e3ad1 (DEFAULT_TABLE table=test-table [id=7efcd0c260ef4180a369c8821c46e077]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:47.622280 2267 tablet_service.cc:1467] Processing CreateTablet for tablet e4b16f5e3f73443eb4c67f4eb81e3ad1 (DEFAULT_TABLE table=test-table [id=7efcd0c260ef4180a369c8821c46e077]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:47.622480 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b16f5e3f73443eb4c67f4eb81e3ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:47.622501 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b16f5e3f73443eb4c67f4eb81e3ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:47.622503 2682 ts_tablet_manager.cc:1918] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:24:47.622583 2682 log.cc:1196] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/6e0f2a86fa7144f6b776228dd371f553
I20260619 02:24:47.622905 2682 ts_tablet_manager.cc:1939] T 6e0f2a86fa7144f6b776228dd371f553 P b9cbec7e31164b49b493d60dc7258845: Deleting consensus metadata
I20260619 02:24:47.624541 2637 tablet_bootstrap.cc:492] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20: Bootstrap starting.
I20260619 02:24:47.624583 2636 tablet_bootstrap.cc:492] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845: Bootstrap starting.
I20260619 02:24:47.625484 2637 tablet_bootstrap.cc:654] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:47.625510 2636 tablet_bootstrap.cc:654] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:47.626217 2114 catalog_manager.cc:4841] TS b9cbec7e31164b49b493d60dc7258845 (127.0.98.2:38529): tablet 6e0f2a86fa7144f6b776228dd371f553 (table test-table [id=58ad9352b118430eb80e5156d8c4ab6a]) successfully deleted
I20260619 02:24:47.626609 2637 tablet_bootstrap.cc:492] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20: No bootstrap required, opened a new log
I20260619 02:24:47.626698 2637 ts_tablet_manager.cc:1397] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:24:47.626674 2547 tablet_service.cc:1467] Processing CreateTablet for tablet e4b16f5e3f73443eb4c67f4eb81e3ad1 (DEFAULT_TABLE table=test-table [id=7efcd0c260ef4180a369c8821c46e077]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:47.626935 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b16f5e3f73443eb4c67f4eb81e3ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:47.627128 2637 raft_consensus.cc:348] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.627249 2637 raft_consensus.cc:374] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:47.627286 2637 raft_consensus.cc:729] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02258e2a40a346ea80523911c8542b20, State: Initialized, Role: FOLLOWER
I20260619 02:24:47.627399 2637 consensus_queue.cc:260] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.627660 2637 ts_tablet_manager.cc:1428] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:24:47.629201 2680 ts_tablet_manager.cc:1918] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:24:47.629240 2638 tablet_bootstrap.cc:492] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b: Bootstrap starting.
I20260619 02:24:47.629352 2680 log.cc:1196] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/6e0f2a86fa7144f6b776228dd371f553
I20260619 02:24:47.629747 2680 ts_tablet_manager.cc:1939] T 6e0f2a86fa7144f6b776228dd371f553 P 8c44b84db3f34bf294eb7755eb8fb61b: Deleting consensus metadata
I20260619 02:24:47.630354 2638 tablet_bootstrap.cc:654] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:47.631254 2114 catalog_manager.cc:4841] TS 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265): tablet 6e0f2a86fa7144f6b776228dd371f553 (table test-table [id=58ad9352b118430eb80e5156d8c4ab6a]) successfully deleted
I20260619 02:24:47.631711 2638 tablet_bootstrap.cc:492] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b: No bootstrap required, opened a new log
I20260619 02:24:47.631811 2638 ts_tablet_manager.cc:1397] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:24:47.632171 2638 raft_consensus.cc:348] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.632282 2638 raft_consensus.cc:374] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:47.632328 2638 raft_consensus.cc:729] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c44b84db3f34bf294eb7755eb8fb61b, State: Initialized, Role: FOLLOWER
I20260619 02:24:47.632436 2638 consensus_queue.cc:260] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.632689 2638 ts_tablet_manager.cc:1428] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:24:47.633118 2117 catalog_manager.cc:4841] TS 02258e2a40a346ea80523911c8542b20 (127.0.98.1:33287): tablet 6e0f2a86fa7144f6b776228dd371f553 (table test-table [id=58ad9352b118430eb80e5156d8c4ab6a]) successfully deleted
I20260619 02:24:47.633657 2636 tablet_bootstrap.cc:492] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845: No bootstrap required, opened a new log
I20260619 02:24:47.633757 2636 ts_tablet_manager.cc:1397] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260619 02:24:47.634186 2636 raft_consensus.cc:348] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.634307 2636 raft_consensus.cc:374] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:47.634357 2636 raft_consensus.cc:729] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9cbec7e31164b49b493d60dc7258845, State: Initialized, Role: FOLLOWER
I20260619 02:24:47.634513 2636 consensus_queue.cc:260] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.634770 2636 ts_tablet_manager.cc:1428] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:24:47.881634 2667 raft_consensus.cc:482] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:24:47.881796 2667 raft_consensus.cc:504] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.882306 2667 leader_election.cc:288] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02258e2a40a346ea80523911c8542b20 (127.0.98.1:33287), 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265)
I20260619 02:24:47.882767 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b16f5e3f73443eb4c67f4eb81e3ad1" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02258e2a40a346ea80523911c8542b20" is_pre_election: true
I20260619 02:24:47.882786 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b16f5e3f73443eb4c67f4eb81e3ad1" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" is_pre_election: true
I20260619 02:24:47.882903 2287 raft_consensus.cc:2454] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 0.
I20260619 02:24:47.882969 2567 raft_consensus.cc:2454] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 0.
I20260619 02:24:47.883265 2361 leader_election.cc:302] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02258e2a40a346ea80523911c8542b20, b9cbec7e31164b49b493d60dc7258845; no voters:
I20260619 02:24:47.883445 2667 raft_consensus.cc:2781] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:24:47.883508 2667 raft_consensus.cc:482] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:24:47.883550 2667 raft_consensus.cc:3037] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:47.884505 2667 raft_consensus.cc:504] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.884822 2667 leader_election.cc:288] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 election: Requested vote from peers 02258e2a40a346ea80523911c8542b20 (127.0.98.1:33287), 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265)
I20260619 02:24:47.885134 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b16f5e3f73443eb4c67f4eb81e3ad1" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02258e2a40a346ea80523911c8542b20"
I20260619 02:24:47.885167 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b16f5e3f73443eb4c67f4eb81e3ad1" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c44b84db3f34bf294eb7755eb8fb61b"
I20260619 02:24:47.885265 2287 raft_consensus.cc:3037] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:47.885265 2567 raft_consensus.cc:3037] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:47.886372 2287 raft_consensus.cc:2454] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 1.
I20260619 02:24:47.886372 2567 raft_consensus.cc:2454] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 1.
I20260619 02:24:47.886644 2358 leader_election.cc:302] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c44b84db3f34bf294eb7755eb8fb61b, b9cbec7e31164b49b493d60dc7258845; no voters:
I20260619 02:24:47.886831 2667 raft_consensus.cc:2781] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:47.886934 2667 raft_consensus.cc:686] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [term 1 LEADER]: Becoming Leader. State: Replica: b9cbec7e31164b49b493d60dc7258845, State: Running, Role: LEADER
I20260619 02:24:47.887074 2667 consensus_queue.cc:237] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:47.888630 2129 catalog_manager.cc:5495] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9cbec7e31164b49b493d60dc7258845 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b9cbec7e31164b49b493d60dc7258845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } health_report { overall_health: UNKNOWN } } }
I20260619 02:24:47.926640 2567 raft_consensus.cc:1261] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Refusing update from remote peer b9cbec7e31164b49b493d60dc7258845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:47.926640 2287 raft_consensus.cc:1261] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Refusing update from remote peer b9cbec7e31164b49b493d60dc7258845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:47.927065 2670 consensus_queue.cc:1035] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:24:47.927348 2642 consensus_queue.cc:1035] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:24:47.939226 2662 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 02:24:49.486117 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:24:49.486258 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:24:49.488343 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2: Sending DeleteTablet for 3 replicas of tablet e4b16f5e3f73443eb4c67f4eb81e3ad1
I20260619 02:24:49.488827 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b16f5e3f73443eb4c67f4eb81e3ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40090
I20260619 02:24:49.488921 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b16f5e3f73443eb4c67f4eb81e3ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59438
I20260619 02:24:49.488989 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b16f5e3f73443eb4c67f4eb81e3ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48008
I20260619 02:24:49.489194 2710 tablet_replica.cc:330] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845: stopping tablet replica
I20260619 02:24:49.489307 2711 tablet_replica.cc:330] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b: stopping tablet replica
I20260619 02:24:49.489310 2710 raft_consensus.cc:2229] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:24:49.489521 2710 raft_consensus.cc:2258] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:49.489622 2711 raft_consensus.cc:2229] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:24:49.489727 2711 raft_consensus.cc:2258] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:49.490223 2711 ts_tablet_manager.cc:1905] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:49.491207 2710 ts_tablet_manager.cc:1905] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:49.492595 2711 ts_tablet_manager.cc:1918] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:24:49.492678 2711 log.cc:1196] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/e4b16f5e3f73443eb4c67f4eb81e3ad1
I20260619 02:24:49.492947 2711 ts_tablet_manager.cc:1939] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 8c44b84db3f34bf294eb7755eb8fb61b: Deleting consensus metadata
I20260619 02:24:49.493582 2114 catalog_manager.cc:4841] TS 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265): tablet e4b16f5e3f73443eb4c67f4eb81e3ad1 (table test-table [id=7efcd0c260ef4180a369c8821c46e077]) successfully deleted
I20260619 02:24:49.493849 2710 ts_tablet_manager.cc:1918] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:24:49.493909 2710 log.cc:1196] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/e4b16f5e3f73443eb4c67f4eb81e3ad1
I20260619 02:24:49.494215 2710 ts_tablet_manager.cc:1939] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P b9cbec7e31164b49b493d60dc7258845: Deleting consensus metadata
I20260619 02:24:49.494788 2114 catalog_manager.cc:4841] TS b9cbec7e31164b49b493d60dc7258845 (127.0.98.2:38529): tablet e4b16f5e3f73443eb4c67f4eb81e3ad1 (table test-table [id=7efcd0c260ef4180a369c8821c46e077]) successfully deleted
I20260619 02:24:49.494999 2709 tablet_replica.cc:330] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20: stopping tablet replica
I20260619 02:24:49.495141 2709 raft_consensus.cc:2229] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:24:49.495273 2709 raft_consensus.cc:2258] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:49.495913 2709 ts_tablet_manager.cc:1905] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:49.498023 2709 ts_tablet_manager.cc:1918] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:24:49.498086 2709 log.cc:1196] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/wal/wals/e4b16f5e3f73443eb4c67f4eb81e3ad1
I20260619 02:24:49.498261 2709 ts_tablet_manager.cc:1939] T e4b16f5e3f73443eb4c67f4eb81e3ad1 P 02258e2a40a346ea80523911c8542b20: Deleting consensus metadata
I20260619 02:24:49.498816 2117 catalog_manager.cc:4841] TS 02258e2a40a346ea80523911c8542b20 (127.0.98.1:33287): tablet e4b16f5e3f73443eb4c67f4eb81e3ad1 (table test-table [id=7efcd0c260ef4180a369c8821c46e077]) successfully deleted
Jun 19 02:24:49 dist-test-slave-xkrt krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835889, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:24:49 dist-test-slave-xkrt krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835889, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:49.518118 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:49.506907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.0.98.62:43821)
0619 02:24:49.507133 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:49.507140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:49.507153 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:24:49.507271 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:49.507273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:49.507312 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:49.507396 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:49.508046 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.508578 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.509389 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.509674 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.509752 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:49.515303 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:49.515306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:49.515308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:49.515329 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:49.516664 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:49.517025 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:49.517028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:49.517029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:49.517100 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:49.517316 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:49.517318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:49.517319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:49.517493 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:49.517685 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:49.517896 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 02:24:49.517992 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260619 02:24:49.520773 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:24:49.521216 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:24:49.525643 2547 tablet_service.cc:1467] Processing CreateTablet for tablet fb7b067cc74e40aaac28205fde3d7648 (DEFAULT_TABLE table=test-table [id=b1c870e1f1024b71b28b22e9e6a4f112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:49.525631 2267 tablet_service.cc:1467] Processing CreateTablet for tablet fb7b067cc74e40aaac28205fde3d7648 (DEFAULT_TABLE table=test-table [id=b1c870e1f1024b71b28b22e9e6a4f112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:49.525923 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb7b067cc74e40aaac28205fde3d7648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:49.525933 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb7b067cc74e40aaac28205fde3d7648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:49.525875 2406 tablet_service.cc:1467] Processing CreateTablet for tablet fb7b067cc74e40aaac28205fde3d7648 (DEFAULT_TABLE table=test-table [id=b1c870e1f1024b71b28b22e9e6a4f112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:49.526170 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb7b067cc74e40aaac28205fde3d7648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:49.528167 2720 tablet_bootstrap.cc:492] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845: Bootstrap starting.
I20260619 02:24:49.528354 2721 tablet_bootstrap.cc:492] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b: Bootstrap starting.
I20260619 02:24:49.528775 2722 tablet_bootstrap.cc:492] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20: Bootstrap starting.
I20260619 02:24:49.529191 2720 tablet_bootstrap.cc:654] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:49.529301 2721 tablet_bootstrap.cc:654] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:49.529783 2722 tablet_bootstrap.cc:654] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:49.530339 2720 tablet_bootstrap.cc:492] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845: No bootstrap required, opened a new log
I20260619 02:24:49.530444 2720 ts_tablet_manager.cc:1397] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:24:49.530926 2720 raft_consensus.cc:348] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.531056 2720 raft_consensus.cc:374] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:49.531096 2720 raft_consensus.cc:729] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9cbec7e31164b49b493d60dc7258845, State: Initialized, Role: FOLLOWER
I20260619 02:24:49.531244 2720 consensus_queue.cc:260] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.531411 2722 tablet_bootstrap.cc:492] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20: No bootstrap required, opened a new log
I20260619 02:24:49.531503 2722 ts_tablet_manager.cc:1397] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:24:49.531551 2720 ts_tablet_manager.cc:1428] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:24:49.531939 2722 raft_consensus.cc:348] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.532051 2722 raft_consensus.cc:374] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:49.532091 2722 raft_consensus.cc:729] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02258e2a40a346ea80523911c8542b20, State: Initialized, Role: FOLLOWER
I20260619 02:24:49.532222 2722 consensus_queue.cc:260] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.532567 2722 ts_tablet_manager.cc:1428] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:24:49.533115 2721 tablet_bootstrap.cc:492] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b: No bootstrap required, opened a new log
I20260619 02:24:49.533211 2721 ts_tablet_manager.cc:1397] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260619 02:24:49.533841 2721 raft_consensus.cc:348] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.533993 2721 raft_consensus.cc:374] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:49.534047 2721 raft_consensus.cc:729] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c44b84db3f34bf294eb7755eb8fb61b, State: Initialized, Role: FOLLOWER
I20260619 02:24:49.534168 2721 consensus_queue.cc:260] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.534570 2721 ts_tablet_manager.cc:1428] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260619 02:24:49.777311 2726 raft_consensus.cc:482] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:24:49.777465 2726 raft_consensus.cc:504] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.778718 2726 leader_election.cc:288] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9cbec7e31164b49b493d60dc7258845 (127.0.98.2:38529), 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265)
I20260619 02:24:49.783408 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:49.778962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38529 (local address 127.0.98.1:41761)
0619 02:24:49.779296 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:49.779315 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:49.779445 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:49.779903 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:49.779906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:49.779932 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:49.780065 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:49.780074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.781082 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.781085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:49.782202 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:49.782212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.782972 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.782975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:49.783071 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:49.783095 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:24:49.783165 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:24:49.783242 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":36,"thread_start_us":134,"threads_started":1}
I20260619 02:24:49.783457 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:49.779079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41761 (local address 127.0.98.2:38529)
0619 02:24:49.779652 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:49.779661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:49.779677 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:24:49.779747 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:49.779749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:49.779789 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:49.779875 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:49.780178 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.780988 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.782329 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.782850 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.782961 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:49.783024 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:49.783213 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 02:24:49.783294 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":127,"threads_started":1}
I20260619 02:24:49.783921 2427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb7b067cc74e40aaac28205fde3d7648" candidate_uuid: "02258e2a40a346ea80523911c8542b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9cbec7e31164b49b493d60dc7258845" is_pre_election: true
I20260619 02:24:49.784147 2427 raft_consensus.cc:2454] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02258e2a40a346ea80523911c8542b20 in term 0.
I20260619 02:24:49.784253 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:49.779365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39265 (local address 127.0.98.1:45283)
0619 02:24:49.779702 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:49.779717 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:49.779804 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:49.780261 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:49.780264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:49.780278 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:24:49.780408 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:49.780412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.781324 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.781328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:49.782594 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:49.782605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.783832 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.783836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:49.783932 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:49.783950 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 02:24:49.783994 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 02:24:49.784074 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":2,"thread_start_us":67,"threads_started":1}
I20260619 02:24:49.784686 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:49.779451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45283 (local address 127.0.98.3:39265)
0619 02:24:49.780025 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:49.780035 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:49.780049 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:24:49.780099 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:49.780105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:49.780178 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:49.780254 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:24:49.780518 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.781150 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.782727 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.783367 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.784427 (+ 1060us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:49.784490 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:49.784532 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 02:24:49.784583 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":128,"threads_started":1}
I20260619 02:24:49.784636 2218 leader_election.cc:302] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02258e2a40a346ea80523911c8542b20, b9cbec7e31164b49b493d60dc7258845; no voters:
I20260619 02:24:49.784963 2726 raft_consensus.cc:2781] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:24:49.785032 2726 raft_consensus.cc:482] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:24:49.785075 2726 raft_consensus.cc:3037] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:49.785032 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb7b067cc74e40aaac28205fde3d7648" candidate_uuid: "02258e2a40a346ea80523911c8542b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" is_pre_election: true
I20260619 02:24:49.785153 2567 raft_consensus.cc:2454] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02258e2a40a346ea80523911c8542b20 in term 0.
I20260619 02:24:49.786217 2726 raft_consensus.cc:504] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.786556 2726 leader_election.cc:288] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [CANDIDATE]: Term 1 election: Requested vote from peers b9cbec7e31164b49b493d60dc7258845 (127.0.98.2:38529), 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265)
I20260619 02:24:49.786885 2427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb7b067cc74e40aaac28205fde3d7648" candidate_uuid: "02258e2a40a346ea80523911c8542b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9cbec7e31164b49b493d60dc7258845"
I20260619 02:24:49.786938 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb7b067cc74e40aaac28205fde3d7648" candidate_uuid: "02258e2a40a346ea80523911c8542b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c44b84db3f34bf294eb7755eb8fb61b"
I20260619 02:24:49.787001 2427 raft_consensus.cc:3037] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:49.787031 2567 raft_consensus.cc:3037] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:49.788100 2567 raft_consensus.cc:2454] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02258e2a40a346ea80523911c8542b20 in term 1.
I20260619 02:24:49.788209 2427 raft_consensus.cc:2454] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02258e2a40a346ea80523911c8542b20 in term 1.
I20260619 02:24:49.788432 2218 leader_election.cc:302] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02258e2a40a346ea80523911c8542b20, 8c44b84db3f34bf294eb7755eb8fb61b; no voters:
I20260619 02:24:49.788599 2726 raft_consensus.cc:2781] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:49.788795 2726 raft_consensus.cc:686] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [term 1 LEADER]: Becoming Leader. State: Replica: 02258e2a40a346ea80523911c8542b20, State: Running, Role: LEADER
I20260619 02:24:49.789040 2726 consensus_queue.cc:237] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.791306 2128 catalog_manager.cc:5495] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02258e2a40a346ea80523911c8542b20 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "02258e2a40a346ea80523911c8542b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } health_report { overall_health: UNKNOWN } } }
I20260619 02:24:49.829708 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:49.826337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.0.98.1:33287)
0619 02:24:49.826589 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:49.826596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:49.826607 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:24:49.826673 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:49.826674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:49.826724 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:49.826824 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:24:49.827016 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.827554 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.828358 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.828581 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.828655 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:49.828894 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:24:49.829335 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:24:49.829422 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:49.829485 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 02:24:49.829528 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":101,"threads_started":1}
I20260619 02:24:49.838971 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:49.835940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38529 (local address 127.0.98.1:52459)
0619 02:24:49.836218 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:49.836231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:49.836313 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:49.836523 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:49.836524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:49.836530 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:24:49.836624 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:49.836628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.837275 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.837277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:49.837984 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:49.837993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.838394 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.838396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:49.838469 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:49.838506 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:24:49.838738 (+ 232us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:24:49.838742 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:24:49.838793 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 02:24:49.838851 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
I20260619 02:24:49.838972 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:49.836078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52459 (local address 127.0.98.2:38529)
0619 02:24:49.836220 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:49.836223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:49.836300 (+ 77us) server_negotiation.cc:407] Connection header received
0619 02:24:49.836395 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:49.836398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:49.836437 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:49.836498 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:24:49.836710 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.837189 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.838082 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.838285 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.838365 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:49.838611 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:24:49.838668 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:24:49.838720 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:49.838834 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 02:24:49.838878 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:24:49.844194 2567 raft_consensus.cc:1261] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Refusing update from remote peer 02258e2a40a346ea80523911c8542b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:49.844401 2427 raft_consensus.cc:1261] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Refusing update from remote peer 02258e2a40a346ea80523911c8542b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:49.844678 2732 consensus_queue.cc:1035] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:24:49.844857 2726 consensus_queue.cc:1035] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:24:49.861572 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:49.858286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38187 (local address 127.0.98.1:33287)
0619 02:24:49.858399 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:49.858403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:49.858641 (+ 238us) server_negotiation.cc:407] Connection header received
0619 02:24:49.858761 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:49.858764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:49.858804 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:49.858899 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:24:49.859122 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.859620 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.860454 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.860753 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.860885 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:49.861113 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:24:49.861190 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:24:49.861283 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:49.861377 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 02:24:49.861422 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:24:49.861596 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:49.858240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33287 (local address 127.0.98.2:38187)
0619 02:24:49.858550 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:49.858561 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:49.858677 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:49.858906 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:49.858909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:49.858915 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:24:49.859012 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:49.859017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.859729 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.859734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:49.860327 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:49.860334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:49.860875 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:49.860879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:49.860963 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:49.860990 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:24:49.861288 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:24:49.861296 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:24:49.861368 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 02:24:49.861447 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":113,"threads_started":1}
I20260619 02:24:49.883416 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:24:49.883558 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:24:49.885514 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2: Sending DeleteTablet for 3 replicas of tablet fb7b067cc74e40aaac28205fde3d7648
I20260619 02:24:49.886086 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet fb7b067cc74e40aaac28205fde3d7648 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40090
I20260619 02:24:49.886169 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet fb7b067cc74e40aaac28205fde3d7648 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48008
I20260619 02:24:49.886251 2709 tablet_replica.cc:330] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20: stopping tablet replica
I20260619 02:24:49.886278 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet fb7b067cc74e40aaac28205fde3d7648 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59438
I20260619 02:24:49.886355 2709 raft_consensus.cc:2229] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:24:49.886343 2711 tablet_replica.cc:330] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b: stopping tablet replica
I20260619 02:24:49.886421 2710 tablet_replica.cc:330] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845: stopping tablet replica
I20260619 02:24:49.886474 2711 raft_consensus.cc:2229] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:24:49.886513 2710 raft_consensus.cc:2229] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:24:49.886572 2711 raft_consensus.cc:2258] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:49.886600 2710 raft_consensus.cc:2258] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:49.886610 2709 raft_consensus.cc:2258] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:49.887336 2711 ts_tablet_manager.cc:1905] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:49.887348 2709 ts_tablet_manager.cc:1905] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:49.887419 2710 ts_tablet_manager.cc:1905] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:49.889760 2711 ts_tablet_manager.cc:1918] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:24:49.889760 2710 ts_tablet_manager.cc:1918] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:24:49.889827 2710 log.cc:1196] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/fb7b067cc74e40aaac28205fde3d7648
I20260619 02:24:49.889827 2711 log.cc:1196] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/fb7b067cc74e40aaac28205fde3d7648
I20260619 02:24:49.890080 2710 ts_tablet_manager.cc:1939] T fb7b067cc74e40aaac28205fde3d7648 P b9cbec7e31164b49b493d60dc7258845: Deleting consensus metadata
I20260619 02:24:49.890080 2711 ts_tablet_manager.cc:1939] T fb7b067cc74e40aaac28205fde3d7648 P 8c44b84db3f34bf294eb7755eb8fb61b: Deleting consensus metadata
I20260619 02:24:49.890579 2709 ts_tablet_manager.cc:1918] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:24:49.890655 2709 log.cc:1196] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-0/wal/wals/fb7b067cc74e40aaac28205fde3d7648
I20260619 02:24:49.890897 2709 ts_tablet_manager.cc:1939] T fb7b067cc74e40aaac28205fde3d7648 P 02258e2a40a346ea80523911c8542b20: Deleting consensus metadata
I20260619 02:24:49.891587 2117 catalog_manager.cc:4841] TS 02258e2a40a346ea80523911c8542b20 (127.0.98.1:33287): tablet fb7b067cc74e40aaac28205fde3d7648 (table test-table [id=b1c870e1f1024b71b28b22e9e6a4f112]) successfully deleted
I20260619 02:24:49.891583 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:24:49.892066 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:24:49.892513 2114 catalog_manager.cc:4841] TS 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265): tablet fb7b067cc74e40aaac28205fde3d7648 (table test-table [id=b1c870e1f1024b71b28b22e9e6a4f112]) successfully deleted
I20260619 02:24:49.892906 2114 catalog_manager.cc:4841] TS b9cbec7e31164b49b493d60dc7258845 (127.0.98.2:38529): tablet fb7b067cc74e40aaac28205fde3d7648 (table test-table [id=b1c870e1f1024b71b28b22e9e6a4f112]) successfully deleted
I20260619 02:24:49.896450 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 62155ac660804ea88ac0895d4cc5e00d (DEFAULT_TABLE table=test-table [id=20ac6221b6d84654baee0181176d3032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:49.896590 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 62155ac660804ea88ac0895d4cc5e00d (DEFAULT_TABLE table=test-table [id=20ac6221b6d84654baee0181176d3032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:49.896711 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62155ac660804ea88ac0895d4cc5e00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:49.896811 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62155ac660804ea88ac0895d4cc5e00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:49.898890 2720 tablet_bootstrap.cc:492] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845: Bootstrap starting.
I20260619 02:24:49.898897 2722 tablet_bootstrap.cc:492] T 62155ac660804ea88ac0895d4cc5e00d P 02258e2a40a346ea80523911c8542b20: Bootstrap starting.
I20260619 02:24:49.899045 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 62155ac660804ea88ac0895d4cc5e00d (DEFAULT_TABLE table=test-table [id=20ac6221b6d84654baee0181176d3032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:24:49.899253 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62155ac660804ea88ac0895d4cc5e00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:49.899668 2720 tablet_bootstrap.cc:654] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:49.899751 2722 tablet_bootstrap.cc:654] T 62155ac660804ea88ac0895d4cc5e00d P 02258e2a40a346ea80523911c8542b20: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:49.900960 2720 tablet_bootstrap.cc:492] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845: No bootstrap required, opened a new log
I20260619 02:24:49.901047 2720 ts_tablet_manager.cc:1397] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:24:49.901298 2721 tablet_bootstrap.cc:492] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b: Bootstrap starting.
I20260619 02:24:49.901403 2720 raft_consensus.cc:348] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.901835 2720 raft_consensus.cc:374] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:49.901887 2720 raft_consensus.cc:729] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9cbec7e31164b49b493d60dc7258845, State: Initialized, Role: FOLLOWER
I20260619 02:24:49.902019 2720 consensus_queue.cc:260] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.902271 2720 ts_tablet_manager.cc:1428] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:24:49.902384 2721 tablet_bootstrap.cc:654] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:49.902678 2722 tablet_bootstrap.cc:492] T 62155ac660804ea88ac0895d4cc5e00d P 02258e2a40a346ea80523911c8542b20: No bootstrap required, opened a new log
I20260619 02:24:49.902745 2722 ts_tablet_manager.cc:1397] T 62155ac660804ea88ac0895d4cc5e00d P 02258e2a40a346ea80523911c8542b20: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 02:24:49.903128 2722 raft_consensus.cc:348] T 62155ac660804ea88ac0895d4cc5e00d P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.903252 2722 raft_consensus.cc:374] T 62155ac660804ea88ac0895d4cc5e00d P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:49.903290 2722 raft_consensus.cc:729] T 62155ac660804ea88ac0895d4cc5e00d P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02258e2a40a346ea80523911c8542b20, State: Initialized, Role: FOLLOWER
I20260619 02:24:49.903399 2722 consensus_queue.cc:260] T 62155ac660804ea88ac0895d4cc5e00d P 02258e2a40a346ea80523911c8542b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.903451 2721 tablet_bootstrap.cc:492] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b: No bootstrap required, opened a new log
I20260619 02:24:49.903540 2721 ts_tablet_manager.cc:1397] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:24:49.903630 2722 ts_tablet_manager.cc:1428] T 62155ac660804ea88ac0895d4cc5e00d P 02258e2a40a346ea80523911c8542b20: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:24:49.903927 2721 raft_consensus.cc:348] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.904033 2721 raft_consensus.cc:374] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:49.904071 2721 raft_consensus.cc:729] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c44b84db3f34bf294eb7755eb8fb61b, State: Initialized, Role: FOLLOWER
I20260619 02:24:49.904201 2721 consensus_queue.cc:260] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:49.904467 2721 ts_tablet_manager.cc:1428] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:24:50.049711 2697 raft_consensus.cc:482] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:24:50.049844 2697 raft_consensus.cc:504] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:50.050307 2697 leader_election.cc:288] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02258e2a40a346ea80523911c8542b20 (127.0.98.1:33287), 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265)
I20260619 02:24:50.050832 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62155ac660804ea88ac0895d4cc5e00d" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02258e2a40a346ea80523911c8542b20" is_pre_election: true
I20260619 02:24:50.050832 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62155ac660804ea88ac0895d4cc5e00d" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" is_pre_election: true
I20260619 02:24:50.050992 2287 raft_consensus.cc:2454] T 62155ac660804ea88ac0895d4cc5e00d P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 0.
I20260619 02:24:50.050992 2567 raft_consensus.cc:2454] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 0.
I20260619 02:24:50.051299 2358 leader_election.cc:302] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c44b84db3f34bf294eb7755eb8fb61b, b9cbec7e31164b49b493d60dc7258845; no voters:
I20260619 02:24:50.051501 2697 raft_consensus.cc:2781] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:24:50.051565 2697 raft_consensus.cc:482] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:24:50.051604 2697 raft_consensus.cc:3037] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:50.052587 2697 raft_consensus.cc:504] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:50.052886 2697 leader_election.cc:288] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 election: Requested vote from peers 02258e2a40a346ea80523911c8542b20 (127.0.98.1:33287), 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265)
I20260619 02:24:50.053187 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62155ac660804ea88ac0895d4cc5e00d" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02258e2a40a346ea80523911c8542b20"
I20260619 02:24:50.053224 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62155ac660804ea88ac0895d4cc5e00d" candidate_uuid: "b9cbec7e31164b49b493d60dc7258845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c44b84db3f34bf294eb7755eb8fb61b"
I20260619 02:24:50.053352 2287 raft_consensus.cc:3037] T 62155ac660804ea88ac0895d4cc5e00d P 02258e2a40a346ea80523911c8542b20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:50.053382 2567 raft_consensus.cc:3037] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:50.054571 2567 raft_consensus.cc:2454] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 1.
I20260619 02:24:50.054606 2287 raft_consensus.cc:2454] T 62155ac660804ea88ac0895d4cc5e00d P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9cbec7e31164b49b493d60dc7258845 in term 1.
I20260619 02:24:50.054858 2358 leader_election.cc:302] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c44b84db3f34bf294eb7755eb8fb61b, b9cbec7e31164b49b493d60dc7258845; no voters:
I20260619 02:24:50.055025 2697 raft_consensus.cc:2781] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:50.055112 2697 raft_consensus.cc:686] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [term 1 LEADER]: Becoming Leader. State: Replica: b9cbec7e31164b49b493d60dc7258845, State: Running, Role: LEADER
I20260619 02:24:50.055269 2697 consensus_queue.cc:237] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } }
I20260619 02:24:50.056814 2129 catalog_manager.cc:5495] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9cbec7e31164b49b493d60dc7258845 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b9cbec7e31164b49b493d60dc7258845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9cbec7e31164b49b493d60dc7258845" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 } health_report { overall_health: UNKNOWN } } }
I20260619 02:24:50.102597 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:50.099453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.0.98.2:38529)
0619 02:24:50.099594 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:50.099599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:50.099658 (+ 59us) server_negotiation.cc:407] Connection header received
0619 02:24:50.099819 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:50.099822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:50.099867 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:50.099955 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:24:50.100198 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:50.100716 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:50.101579 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:50.101803 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:50.101917 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:50.102167 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:24:50.102245 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:24:50.102324 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:50.102410 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 02:24:50.102455 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:24:50.108175 2287 raft_consensus.cc:1261] T 62155ac660804ea88ac0895d4cc5e00d P 02258e2a40a346ea80523911c8542b20 [term 1 FOLLOWER]: Refusing update from remote peer b9cbec7e31164b49b493d60dc7258845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:50.108306 2567 raft_consensus.cc:1261] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Refusing update from remote peer b9cbec7e31164b49b493d60dc7258845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:24:50.108625 2689 consensus_queue.cc:1035] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02258e2a40a346ea80523911c8542b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:24:50.108774 2697 consensus_queue.cc:1035] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c44b84db3f34bf294eb7755eb8fb61b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:24:50.120891 2690 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 02:24:51.667516 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:24:51.667661 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:24:51.669863 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af1e68eda2c7480d9d1b1bcc2ce214f2: Sending DeleteTablet for 3 replicas of tablet 62155ac660804ea88ac0895d4cc5e00d
I20260619 02:24:51.670475 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 62155ac660804ea88ac0895d4cc5e00d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40090
I20260619 02:24:51.670604 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet 62155ac660804ea88ac0895d4cc5e00d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59438
I20260619 02:24:51.670614 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet 62155ac660804ea88ac0895d4cc5e00d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:24:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48008
I20260619 02:24:51.670886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 2202
I20260619 02:24:51.670944 2773 tablet_replica.cc:330] T 62155ac660804ea88ac0895d4cc5e00d P 02258e2a40a346ea80523911c8542b20: stopping tablet replica
I20260619 02:24:51.672619 2774 tablet_replica.cc:330] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845: stopping tablet replica
I20260619 02:24:51.672777 2774 raft_consensus.cc:2229] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:24:51.672971 2774 raft_consensus.cc:2258] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:51.673238 2775 tablet_replica.cc:330] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b: stopping tablet replica
I20260619 02:24:51.673367 2775 raft_consensus.cc:2229] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:24:51.673503 2775 raft_consensus.cc:2258] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:24:51.674077 2774 ts_tablet_manager.cc:1905] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:51.674134 2775 ts_tablet_manager.cc:1905] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:24:51.676594 2775 ts_tablet_manager.cc:1918] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:24:51.676627 2774 ts_tablet_manager.cc:1918] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:24:51.676712 2774 log.cc:1196] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/62155ac660804ea88ac0895d4cc5e00d
I20260619 02:24:51.676720 2775 log.cc:1196] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/62155ac660804ea88ac0895d4cc5e00d
I20260619 02:24:51.676983 2775 ts_tablet_manager.cc:1939] T 62155ac660804ea88ac0895d4cc5e00d P 8c44b84db3f34bf294eb7755eb8fb61b: Deleting consensus metadata
I20260619 02:24:51.677100 2774 ts_tablet_manager.cc:1939] T 62155ac660804ea88ac0895d4cc5e00d P b9cbec7e31164b49b493d60dc7258845: Deleting consensus metadata
I20260619 02:24:51.677646 2114 catalog_manager.cc:4841] TS 8c44b84db3f34bf294eb7755eb8fb61b (127.0.98.3:39265): tablet 62155ac660804ea88ac0895d4cc5e00d (table test-table [id=20ac6221b6d84654baee0181176d3032]) successfully deleted
I20260619 02:24:51.677794 2114 catalog_manager.cc:4841] TS b9cbec7e31164b49b493d60dc7258845 (127.0.98.2:38529): tablet 62155ac660804ea88ac0895d4cc5e00d (table test-table [id=20ac6221b6d84654baee0181176d3032]) successfully deleted
W20260619 02:24:51.678682 2117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33287 (error 108)
I20260619 02:24:51.679021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 2342
I20260619 02:24:51.682307 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:51.681823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33287 (local address 127.0.0.1:40102)
0619 02:24:51.682094 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:51.682182 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":79,"threads_started":1}
W20260619 02:24:51.682472 2117 catalog_manager.cc:4584] TS 02258e2a40a346ea80523911c8542b20 (127.0.98.1:33287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62155ac660804ea88ac0895d4cc5e00d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33287: connect: Connection refused (error 111)
I20260619 02:24:51.686235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 2482
I20260619 02:24:51.691727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 2098
2026-06-19T02:24:51Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6741 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:24:51 dist-test-slave-xkrt krb5kdc[2780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:24:51 dist-test-slave-xkrt krb5kdc[2780](info): set up 2 sockets
Jun 19 02:24:51 dist-test-slave-xkrt krb5kdc[2780](info): commencing operation
krb5kdc: starting...
W20260619 02:24:52.737421 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:24:52 dist-test-slave-xkrt krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:24:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:24:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:24:52.856732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36927
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:52.961292 2793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:52.961596 2793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:52.961671 2793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:52.966277 2793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:24:52.966354 2793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:52.966404 2793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:52.966440 2793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:24:52.966476 2793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:24:52.971414 2793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:52.972347 2793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:52.973305 2793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:52.981652 2799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:52.981660 2798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:52.981702 2793 server_base.cc:1034] running on GCE node
W20260619 02:24:52.981664 2801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:52.982383 2793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:52.983400 2793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:52.984580 2793 hybrid_clock.cc:648] HybridClock initialized: now 1781835892984559 us; error 39 us; skew 500 ppm
Jun 19 02:24:52 dist-test-slave-xkrt krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:52.987223 2793 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:24:52.988073 2793 webserver.cc:460] Webserver started at http://127.0.98.62:33723/ using document root <none> and password file <none>
I20260619 02:24:52.988597 2793 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:52.988684 2793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:52.988921 2793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:52.990881 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "88ae21e64e9542fb9049d5f9e0bb592e"
format_stamp: "Formatted at 2026-06-19 02:24:52 on dist-test-slave-xkrt"
server_key: "f2720d2cad5749552f9e0a7d2fa919a8"
server_key_iv: "aeb62d1804bc5bb5d7e59e444f76515c"
server_key_version: "encryptionkey@0"
I20260619 02:24:52.991484 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "88ae21e64e9542fb9049d5f9e0bb592e"
format_stamp: "Formatted at 2026-06-19 02:24:52 on dist-test-slave-xkrt"
server_key: "f2720d2cad5749552f9e0a7d2fa919a8"
server_key_iv: "aeb62d1804bc5bb5d7e59e444f76515c"
server_key_version: "encryptionkey@0"
I20260619 02:24:52.994954 2793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:24:52.997160 2808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:52.998315 2793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:24:52.998453 2793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "88ae21e64e9542fb9049d5f9e0bb592e"
format_stamp: "Formatted at 2026-06-19 02:24:52 on dist-test-slave-xkrt"
server_key: "f2720d2cad5749552f9e0a7d2fa919a8"
server_key_iv: "aeb62d1804bc5bb5d7e59e444f76515c"
server_key_version: "encryptionkey@0"
I20260619 02:24:52.998559 2793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:53.010970 2793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:53.013599 2793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:53.013777 2793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:53.020252 2793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36927
I20260619 02:24:53.020278 2860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36927 every 8 connection(s)
I20260619 02:24:53.021136 2793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:24:53.021855 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 2793
I20260619 02:24:53.021938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:24:53.022205 392 external_mini_cluster.cc:1428] Setting key d8582706877d637f05b4205705833382
I20260619 02:24:53.024216 2861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:53.030673 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e: Bootstrap starting.
Jun 19 02:24:53 dist-test-slave-xkrt krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:53.033058 2861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:53.033984 2861 log.cc:824] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:53.036127 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e: No bootstrap required, opened a new log
I20260619 02:24:53.037312 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:53.023621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45218 (local address 127.0.98.62:36927)
0619 02:24:53.024051 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:53.024069 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:53.024110 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:24:53.024932 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:53.024954 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:53.025340 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:53.025720 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:53.026378 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:53.027850 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:53.028716 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:53.029401 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:53.029598 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:53.032156 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:53.032162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:53.032175 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:53.032203 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:53.034175 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:53.034654 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:53.034658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:53.034662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:53.034808 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:53.035069 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:53.035071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:53.035073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:53.035537 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:53.035723 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:53.035917 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 02:24:53.036193 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":128,"threads_started":1}
I20260619 02:24:53.039618 2861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ae21e64e9542fb9049d5f9e0bb592e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36927 } }
I20260619 02:24:53.039815 2861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:53.039848 2861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ae21e64e9542fb9049d5f9e0bb592e, State: Initialized, Role: FOLLOWER
I20260619 02:24:53.040179 2861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ae21e64e9542fb9049d5f9e0bb592e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36927 } }
I20260619 02:24:53.040280 2861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:24:53.040318 2861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:24:53.040383 2861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:53.041123 2861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ae21e64e9542fb9049d5f9e0bb592e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36927 } }
I20260619 02:24:53.041390 2861 leader_election.cc:302] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88ae21e64e9542fb9049d5f9e0bb592e; no voters:
I20260619 02:24:53.041677 2861 leader_election.cc:288] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:24:53.041816 2866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:53.042064 2866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [term 1 LEADER]: Becoming Leader. State: Replica: 88ae21e64e9542fb9049d5f9e0bb592e, State: Running, Role: LEADER
I20260619 02:24:53.042369 2866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ae21e64e9542fb9049d5f9e0bb592e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36927 } }
I20260619 02:24:53.042771 2861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:24:53.043887 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88ae21e64e9542fb9049d5f9e0bb592e. Latest consensus state: current_term: 1 leader_uuid: "88ae21e64e9542fb9049d5f9e0bb592e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ae21e64e9542fb9049d5f9e0bb592e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36927 } } }
I20260619 02:24:53.044006 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:53.044152 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88ae21e64e9542fb9049d5f9e0bb592e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ae21e64e9542fb9049d5f9e0bb592e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36927 } } }
I20260619 02:24:53.044318 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e [sys.catalog]: This master's current role is: LEADER
W20260619 02:24:53.047214 2881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:24:53.047278 2881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:24:53.047359 2877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:24:53.048035 2877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:24:53.052951 2877 catalog_manager.cc:1347] Generated new cluster ID: 7722d15d53b0446e952ba972a98f5861
I20260619 02:24:53.053011 2877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:24:53.064426 2877 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:24:53.065721 2877 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:24:53.084542 2877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88ae21e64e9542fb9049d5f9e0bb592e: Generated new TSK 0
I20260619 02:24:53.085489 2877 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:24:53.120844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36927
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:53.228487 2887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:53.228722 2887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:53.228761 2887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:53.232689 2887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:53.232738 2887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:53.232831 2887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:24:53.237776 2887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:53.238711 2887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:53.239681 2887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:53.248896 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:53.248903 2892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:53.249012 2887 server_base.cc:1034] running on GCE node
W20260619 02:24:53.248883 2895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:53.249467 2887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:53.250082 2887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:53.251230 2887 hybrid_clock.cc:648] HybridClock initialized: now 1781835893251203 us; error 42 us; skew 500 ppm
Jun 19 02:24:53 dist-test-slave-xkrt krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:53.253785 2887 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:24:53.254549 2887 webserver.cc:460] Webserver started at http://127.0.98.1:40563/ using document root <none> and password file <none>
I20260619 02:24:53.255069 2887 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:53.255127 2887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:53.255343 2887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:53.257206 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "4a0c37667208415e868b6fafec4186a9"
format_stamp: "Formatted at 2026-06-19 02:24:53 on dist-test-slave-xkrt"
server_key: "23a74deea8e5cc8819f13d4dd892e20c"
server_key_iv: "135e570bf22ea6a6c23792c24259b20f"
server_key_version: "encryptionkey@0"
I20260619 02:24:53.257768 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4a0c37667208415e868b6fafec4186a9"
format_stamp: "Formatted at 2026-06-19 02:24:53 on dist-test-slave-xkrt"
server_key: "23a74deea8e5cc8819f13d4dd892e20c"
server_key_iv: "135e570bf22ea6a6c23792c24259b20f"
server_key_version: "encryptionkey@0"
I20260619 02:24:53.261096 2887 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:24:53.263695 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:53.264842 2887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:24:53.264994 2887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "4a0c37667208415e868b6fafec4186a9"
format_stamp: "Formatted at 2026-06-19 02:24:53 on dist-test-slave-xkrt"
server_key: "23a74deea8e5cc8819f13d4dd892e20c"
server_key_iv: "135e570bf22ea6a6c23792c24259b20f"
server_key_version: "encryptionkey@0"
I20260619 02:24:53.265098 2887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:53.281340 2887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:53.284431 2887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:53.284621 2887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:53.285164 2887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:53.285995 2887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:53.286063 2887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:53.286134 2887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:53.286180 2887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:53.294569 2887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35969
I20260619 02:24:53.294628 3015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35969 every 8 connection(s)
I20260619 02:24:53.295441 2887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:24:53.296245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 2887
I20260619 02:24:53.296363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:24:53.296633 392 external_mini_cluster.cc:1428] Setting key 098d67c482cfe6a233db1767f2b8c826
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 19 02:24:53 dist-test-slave-xkrt krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 02:24:53.310559 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:53.297386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56745 (local address 127.0.98.62:36927)
0619 02:24:53.297507 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:53.297514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:53.298746 (+ 1232us) server_negotiation.cc:407] Connection header received
0619 02:24:53.300237 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:53.300243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:53.300306 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:53.300406 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:53.301745 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:53.302318 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:53.303765 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:53.304011 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:53.304125 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:53.306838 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:53.306841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:53.306843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:53.306867 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:53.308313 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:53.308799 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:53.308808 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:53.308810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:53.308886 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:53.309287 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:53.309292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:53.309294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:53.309571 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:53.309682 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:53.310183 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 02:24:53.310328 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:24:53.311478 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:53.297906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36927 (local address 127.0.98.1:56745)
0619 02:24:53.298563 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:53.298615 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:53.300026 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:53.300548 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:53.300554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:53.300898 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:53.301571 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:53.301590 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:53.302522 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:53.302525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:53.303631 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:53.303640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:53.304159 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:53.304163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:53.304346 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:53.304996 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:53.305012 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:53.306681 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:53.308425 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:53.308429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:53.308441 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:53.308684 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:53.309000 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:53.309004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:53.309006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:53.309142 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:53.309683 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:53.309688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:53.309931 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 02:24:53.310082 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 02:24:53.310365 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":188,"threads_started":1}
I20260619 02:24:53.312775 3016 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36927
I20260619 02:24:53.313035 3016 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:53.313517 3016 heartbeater.cc:511] Master 127.0.98.62:36927 requested a full tablet report, sending...
I20260619 02:24:53.315035 2825 ts_manager.cc:194] Registered new tserver with Master: 4a0c37667208415e868b6fafec4186a9 (127.0.98.1:35969)
I20260619 02:24:53.316128 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:24:53.324883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36927
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:53.431125 3021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:53.431365 3021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:53.431404 3021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:53.435250 3021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:53.435293 3021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:53.435345 3021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:24:53.440287 3021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:53.441154 3021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:53.442067 3021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:53.451287 3026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:53.451277 3027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:53.451256 3029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:53.451607 3021 server_base.cc:1034] running on GCE node
I20260619 02:24:53.451997 3021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:53.452589 3021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:53.453768 3021 hybrid_clock.cc:648] HybridClock initialized: now 1781835893453737 us; error 43 us; skew 500 ppm
Jun 19 02:24:53 dist-test-slave-xkrt krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:53.456380 3021 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:24:53.457204 3021 webserver.cc:460] Webserver started at http://127.0.98.2:33171/ using document root <none> and password file <none>
I20260619 02:24:53.457826 3021 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:53.457916 3021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:53.458135 3021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:53.460078 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "39cb1f59af2a4c55b73c95ec05d30c50"
format_stamp: "Formatted at 2026-06-19 02:24:53 on dist-test-slave-xkrt"
server_key: "99743325b92985f00ce782b398f2f804"
server_key_iv: "ffb37c4e06ff7dadc2609acdf56af29a"
server_key_version: "encryptionkey@0"
I20260619 02:24:53.460647 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "39cb1f59af2a4c55b73c95ec05d30c50"
format_stamp: "Formatted at 2026-06-19 02:24:53 on dist-test-slave-xkrt"
server_key: "99743325b92985f00ce782b398f2f804"
server_key_iv: "ffb37c4e06ff7dadc2609acdf56af29a"
server_key_version: "encryptionkey@0"
I20260619 02:24:53.464148 3021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 02:24:53.466456 3036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:53.467604 3021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:24:53.467730 3021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "39cb1f59af2a4c55b73c95ec05d30c50"
format_stamp: "Formatted at 2026-06-19 02:24:53 on dist-test-slave-xkrt"
server_key: "99743325b92985f00ce782b398f2f804"
server_key_iv: "ffb37c4e06ff7dadc2609acdf56af29a"
server_key_version: "encryptionkey@0"
I20260619 02:24:53.467839 3021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:53.479446 3021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:53.482188 3021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:53.482388 3021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:53.482913 3021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:53.483680 3021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:53.483750 3021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:53.483819 3021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:53.483866 3021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:53.492107 3021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45363
I20260619 02:24:53.492173 3149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45363 every 8 connection(s)
I20260619 02:24:53.492992 3021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:24:53.500425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 3021
I20260619 02:24:53.500520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:24:53.500772 392 external_mini_cluster.cc:1428] Setting key b35e190f9303afda26cda899b2d8d22e
Jun 19 02:24:53 dist-test-slave-xkrt krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:53.507164 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:53.494684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46231 (local address 127.0.98.62:36927)
0619 02:24:53.494838 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:53.494843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:53.495580 (+ 737us) server_negotiation.cc:407] Connection header received
0619 02:24:53.496888 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:53.496891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:53.496936 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:53.497009 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:53.498152 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:53.498658 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:53.499886 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:53.500088 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:53.500159 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:53.503012 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:53.503016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:53.503019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:53.503045 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:53.505012 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:53.505487 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:53.505493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:53.505496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:53.505624 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:53.505930 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:53.505936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:53.505941 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:53.506205 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:53.506329 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:53.506776 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 02:24:53.506926 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:24:53.507802 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:53.495032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36927 (local address 127.0.98.2:46231)
0619 02:24:53.495439 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:53.495478 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:53.496686 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:53.497165 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:53.497171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:53.497491 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:53.498034 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:53.498047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:53.498797 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:53.498800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:53.499761 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:53.499771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:53.500221 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:53.500224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:53.500384 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:53.501080 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:53.501100 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:53.502851 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:53.505128 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:53.505132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:53.505142 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:53.505385 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:53.505711 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:53.505713 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:53.505714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:53.505833 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:53.506320 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:53.506324 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:53.506539 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 02:24:53.506683 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 02:24:53.506908 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260619 02:24:53.508905 3150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36927
I20260619 02:24:53.509130 3150 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:53.509616 3150 heartbeater.cc:511] Master 127.0.98.62:36927 requested a full tablet report, sending...
I20260619 02:24:53.510627 2825 ts_manager.cc:194] Registered new tserver with Master: 39cb1f59af2a4c55b73c95ec05d30c50 (127.0.98.2:45363)
I20260619 02:24:53.511171 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46231
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:24:53.530578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36927
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:53.638811 3155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:53.639114 3155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:53.639194 3155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:53.643273 3155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:53.643359 3155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:53.643474 3155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:24:53.648838 3155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:53.649899 3155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:53.650952 3155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:53.660627 3160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:53.660668 3155 server_base.cc:1034] running on GCE node
W20260619 02:24:53.660565 3161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:53.660565 3163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:53.661299 3155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:53.662004 3155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:53.663183 3155 hybrid_clock.cc:648] HybridClock initialized: now 1781835893663163 us; error 29 us; skew 500 ppm
Jun 19 02:24:53 dist-test-slave-xkrt krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:53.666198 3155 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:24:53.667157 3155 webserver.cc:460] Webserver started at http://127.0.98.3:35617/ using document root <none> and password file <none>
I20260619 02:24:53.667739 3155 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:53.667834 3155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:53.668084 3155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:53.670269 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "acbc8779a2f844c69c0faa70ce0a9a19"
format_stamp: "Formatted at 2026-06-19 02:24:53 on dist-test-slave-xkrt"
server_key: "a218edd91fa194eee0ca27600d5eb681"
server_key_iv: "96ef9a4d596771c99828caab7e7d2764"
server_key_version: "encryptionkey@0"
I20260619 02:24:53.670874 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "acbc8779a2f844c69c0faa70ce0a9a19"
format_stamp: "Formatted at 2026-06-19 02:24:53 on dist-test-slave-xkrt"
server_key: "a218edd91fa194eee0ca27600d5eb681"
server_key_iv: "96ef9a4d596771c99828caab7e7d2764"
server_key_version: "encryptionkey@0"
I20260619 02:24:53.674799 3155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:24:53.677196 3170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:53.678453 3155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:24:53.678599 3155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "acbc8779a2f844c69c0faa70ce0a9a19"
format_stamp: "Formatted at 2026-06-19 02:24:53 on dist-test-slave-xkrt"
server_key: "a218edd91fa194eee0ca27600d5eb681"
server_key_iv: "96ef9a4d596771c99828caab7e7d2764"
server_key_version: "encryptionkey@0"
I20260619 02:24:53.678714 3155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:53.698890 3155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:53.702448 3155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:53.702651 3155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:53.703317 3155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:53.704227 3155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:53.704310 3155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:53.704381 3155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:53.704437 3155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:53.714089 3155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33161
I20260619 02:24:53.714119 3283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33161 every 8 connection(s)
I20260619 02:24:53.714965 3155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:24:53.716401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 3155
I20260619 02:24:53.716512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:24:53.716830 392 external_mini_cluster.cc:1428] Setting key 8832c7f3358bbec4cae00d4a27749cab
Jun 19 02:24:53 dist-test-slave-xkrt krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:53.728277 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:53.716794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40697 (local address 127.0.98.62:36927)
0619 02:24:53.716931 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:53.716935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:53.717938 (+ 1003us) server_negotiation.cc:407] Connection header received
0619 02:24:53.719396 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:53.719398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:53.719440 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:53.719505 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:53.720626 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:53.721143 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:53.722351 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:53.722562 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:53.722644 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:53.725182 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:53.725185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:53.725187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:53.725208 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:53.726507 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:53.726996 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:53.726998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:53.726999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:53.727064 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:53.727382 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:53.727384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:53.727385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:53.727538 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:53.727609 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:53.728077 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 02:24:53.728166 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:24:53.729108 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:53.717285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36927 (local address 127.0.98.3:40697)
0619 02:24:53.717789 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:53.717843 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:53.719202 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:53.719638 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:53.719644 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:53.719990 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:53.720469 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:53.720482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:53.721299 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:53.721302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:53.722231 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:53.722250 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:53.722690 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:53.722693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:53.722831 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:53.723488 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:53.723505 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:53.724991 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:53.726617 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:53.726622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:53.726632 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:53.726910 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:53.727150 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:53.727152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:53.727153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:53.727301 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:53.727625 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:53.727629 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:53.727869 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 02:24:53.728026 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 02:24:53.728310 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":141,"threads_started":1}
I20260619 02:24:53.730242 3284 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36927
I20260619 02:24:53.730468 3284 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:53.730940 3284 heartbeater.cc:511] Master 127.0.98.62:36927 requested a full tablet report, sending...
I20260619 02:24:53.731915 2825 ts_manager.cc:194] Registered new tserver with Master: acbc8779a2f844c69c0faa70ce0a9a19 (127.0.98.3:33161)
I20260619 02:24:53.732468 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40697
I20260619 02:24:53.742187 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:24:53.754822 2864 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:24:53.753362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45228 (local address 127.0.98.62:36927)
0619 02:24:53.753488 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:53.753492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:53.753730 (+ 238us) server_negotiation.cc:407] Connection header received
0619 02:24:53.753834 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:53.753836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:53.753875 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:53.753958 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:53.754690 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45228: BlockingRecv error: recv got EOF from 127.0.0.1:45228 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:24:53.756006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 2887
I20260619 02:24:53.761312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 3021
I20260619 02:24:53.766296 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 3155
I20260619 02:24:53.771878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 2793
2026-06-19T02:24:53Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2078 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:24:53 dist-test-slave-xkrt krb5kdc[3296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:24:53 dist-test-slave-xkrt krb5kdc[3296](info): set up 2 sockets
Jun 19 02:24:53 dist-test-slave-xkrt krb5kdc[3296](info): commencing operation
krb5kdc: starting...
W20260619 02:24:54.805426 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:24:54 dist-test-slave-xkrt krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:24:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:24:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:24:54.924335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35847
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:55.034104 3309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:55.034361 3309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:55.034404 3309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:55.038372 3309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:24:55.038425 3309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:55.038445 3309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:55.038461 3309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:24:55.038476 3309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:24:55.043607 3309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:55.044533 3309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:55.045457 3309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:55.053751 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:55.053751 3315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:55.053817 3314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:55.053926 3309 server_base.cc:1034] running on GCE node
I20260619 02:24:55.054575 3309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:55.055619 3309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:55.056790 3309 hybrid_clock.cc:648] HybridClock initialized: now 1781835895056768 us; error 31 us; skew 500 ppm
Jun 19 02:24:55 dist-test-slave-xkrt krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:55.064105 3309 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:24:55.065053 3309 webserver.cc:460] Webserver started at http://127.0.98.62:44697/ using document root <none> and password file <none>
I20260619 02:24:55.065693 3309 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:55.065759 3309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:55.065961 3309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:55.068020 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "f6f7355b5c7d4cfaa76c28d0f1f6e8e7"
format_stamp: "Formatted at 2026-06-19 02:24:55 on dist-test-slave-xkrt"
server_key: "c4928a320fcae80a9175afe0a4d34e12"
server_key_iv: "a5e594186785f2d8adf62fc84c38ca57"
server_key_version: "encryptionkey@0"
I20260619 02:24:55.068578 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6f7355b5c7d4cfaa76c28d0f1f6e8e7"
format_stamp: "Formatted at 2026-06-19 02:24:55 on dist-test-slave-xkrt"
server_key: "c4928a320fcae80a9175afe0a4d34e12"
server_key_iv: "a5e594186785f2d8adf62fc84c38ca57"
server_key_version: "encryptionkey@0"
I20260619 02:24:55.072307 3309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:24:55.074847 3324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:55.076196 3309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:24:55.076411 3309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "f6f7355b5c7d4cfaa76c28d0f1f6e8e7"
format_stamp: "Formatted at 2026-06-19 02:24:55 on dist-test-slave-xkrt"
server_key: "c4928a320fcae80a9175afe0a4d34e12"
server_key_iv: "a5e594186785f2d8adf62fc84c38ca57"
server_key_version: "encryptionkey@0"
I20260619 02:24:55.076526 3309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:55.092681 3309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:55.095868 3309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:55.096064 3309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:55.103214 3309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35847
I20260619 02:24:55.103575 3376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35847 every 8 connection(s)
I20260619 02:24:55.104157 3309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:24:55.106789 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:55.109750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 3309
I20260619 02:24:55.109869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:24:55.110138 392 external_mini_cluster.cc:1428] Setting key eeb8a01825e0c220bb5f85ca8ef96438
I20260619 02:24:55.112449 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7: Bootstrap starting.
I20260619 02:24:55.114818 3377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:55.115717 3377 log.cc:824] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:55.117695 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7: No bootstrap required, opened a new log
Jun 19 02:24:55 dist-test-slave-xkrt krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:55.121116 3377 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f7355b5c7d4cfaa76c28d0f1f6e8e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35847 } }
I20260619 02:24:55.121353 3377 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:55.121449 3377 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f7355b5c7d4cfaa76c28d0f1f6e8e7, State: Initialized, Role: FOLLOWER
I20260619 02:24:55.121932 3377 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f7355b5c7d4cfaa76c28d0f1f6e8e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35847 } }
I20260619 02:24:55.122081 3377 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:24:55.122169 3377 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:24:55.122282 3377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:55.123421 3377 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f7355b5c7d4cfaa76c28d0f1f6e8e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35847 } }
I20260619 02:24:55.123755 3377 leader_election.cc:302] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f7355b5c7d4cfaa76c28d0f1f6e8e7; no voters:
I20260619 02:24:55.124049 3377 leader_election.cc:288] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:24:55.124204 3382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:55.124465 3382 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [term 1 LEADER]: Becoming Leader. State: Replica: f6f7355b5c7d4cfaa76c28d0f1f6e8e7, State: Running, Role: LEADER
I20260619 02:24:55.124492 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:55.111351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43418 (local address 127.0.98.62:35847)
0619 02:24:55.111767 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:55.111783 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:55.111814 (+ 31us) server_negotiation.cc:407] Connection header received
0619 02:24:55.112394 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:55.112409 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:55.112771 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:55.113028 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:55.114013 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:55.115040 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:55.116198 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:55.116774 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:55.116888 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:55.119601 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:55.119606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:55.119617 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:55.119640 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:55.121275 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:55.121823 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:55.121834 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:55.121838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:55.121940 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:55.122236 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:55.122239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:55.122241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:55.122786 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:55.122993 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:55.123295 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 02:24:55.123575 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":162,"threads_started":1}
I20260619 02:24:55.124804 3382 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f7355b5c7d4cfaa76c28d0f1f6e8e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35847 } }
I20260619 02:24:55.125190 3377 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:24:55.125883 3382 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6f7355b5c7d4cfaa76c28d0f1f6e8e7. Latest consensus state: current_term: 1 leader_uuid: "f6f7355b5c7d4cfaa76c28d0f1f6e8e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f7355b5c7d4cfaa76c28d0f1f6e8e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35847 } } }
I20260619 02:24:55.125983 3382 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:55.126806 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6f7355b5c7d4cfaa76c28d0f1f6e8e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f7355b5c7d4cfaa76c28d0f1f6e8e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35847 } } }
I20260619 02:24:55.126961 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7 [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:55.127473 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 02:24:55.130254 3397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:24:55.130316 3397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:24:55.130424 3389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:24:55.131240 3389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:24:55.136540 3389 catalog_manager.cc:1347] Generated new cluster ID: 24a46a5f944a4b5e9b22b7630dc49719
I20260619 02:24:55.136595 3389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:24:55.142688 3389 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:24:55.143551 3389 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:24:55.151284 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:55.145808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.0.98.62:35847)
0619 02:24:55.145931 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:55.145936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:55.146143 (+ 207us) server_negotiation.cc:407] Connection header received
0619 02:24:55.146262 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:55.146266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:55.146313 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:55.146404 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:55.146929 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:55.147777 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:55.148917 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:55.149279 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:55.149378 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:55.150313 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:55.150317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:55.150320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:24:55.150868 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:55.150900 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:24:55.150917 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:55.151005 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:55.151086 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 02:24:55.151143 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:24:55.155520 3389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6f7355b5c7d4cfaa76c28d0f1f6e8e7: Generated new TSK 0
I20260619 02:24:55.156165 3389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:24:55.163031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 3309
2026-06-19T02:24:55Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1388 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:24:55 dist-test-slave-xkrt krb5kdc[3410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:24:55 dist-test-slave-xkrt krb5kdc[3410](info): set up 2 sockets
Jun 19 02:24:55 dist-test-slave-xkrt krb5kdc[3410](info): commencing operation
krb5kdc: starting...
W20260619 02:24:56.205605 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:24:56 dist-test-slave-xkrt krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:24:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:24:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:24:56.322613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41495
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:56.427088 3423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:56.427362 3423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:56.427433 3423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:56.431331 3423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:24:56.431416 3423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:56.431452 3423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:56.431519 3423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:24:56.431555 3423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:24:56.436533 3423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:56.437510 3423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:56.438474 3423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:56.446550 3428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:56.446558 3429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:56.446588 3431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:56.446733 3423 server_base.cc:1034] running on GCE node
I20260619 02:24:56.447290 3423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:56.448277 3423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:56.449430 3423 hybrid_clock.cc:648] HybridClock initialized: now 1781835896449411 us; error 32 us; skew 500 ppm
Jun 19 02:24:56 dist-test-slave-xkrt krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:56.452062 3423 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:24:56.452837 3423 webserver.cc:460] Webserver started at http://127.0.98.62:33777/ using document root <none> and password file <none>
I20260619 02:24:56.453361 3423 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:56.453446 3423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:56.453708 3423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:56.455608 3423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "6bf1bb1ab000444fbdc3e95b38eae751"
format_stamp: "Formatted at 2026-06-19 02:24:56 on dist-test-slave-xkrt"
server_key: "f29f54f98797049d68503599f98ab3fc"
server_key_iv: "3788ff5748e816df51f1f83073b1d9b2"
server_key_version: "encryptionkey@0"
I20260619 02:24:56.456157 3423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "6bf1bb1ab000444fbdc3e95b38eae751"
format_stamp: "Formatted at 2026-06-19 02:24:56 on dist-test-slave-xkrt"
server_key: "f29f54f98797049d68503599f98ab3fc"
server_key_iv: "3788ff5748e816df51f1f83073b1d9b2"
server_key_version: "encryptionkey@0"
I20260619 02:24:56.459607 3423 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 02:24:56.461961 3438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:56.463183 3423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:24:56.463333 3423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "6bf1bb1ab000444fbdc3e95b38eae751"
format_stamp: "Formatted at 2026-06-19 02:24:56 on dist-test-slave-xkrt"
server_key: "f29f54f98797049d68503599f98ab3fc"
server_key_iv: "3788ff5748e816df51f1f83073b1d9b2"
server_key_version: "encryptionkey@0"
I20260619 02:24:56.463441 3423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:56.478484 3423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:56.481230 3423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:56.481426 3423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:56.487860 3423 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41495
I20260619 02:24:56.488123 3490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41495 every 8 connection(s)
I20260619 02:24:56.488766 3423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:24:56.491293 3491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:56.496238 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751: Bootstrap starting.
I20260619 02:24:56.497464 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 3423
I20260619 02:24:56.497598 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:24:56.497885 392 external_mini_cluster.cc:1428] Setting key d8b57ed3adbd2eb7427a1fb3d3a099d6
I20260619 02:24:56.498831 3491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:56.499565 3491 log.cc:824] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:56.501209 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751: No bootstrap required, opened a new log
I20260619 02:24:56.504464 3491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf1bb1ab000444fbdc3e95b38eae751" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41495 } }
I20260619 02:24:56.504702 3491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:56.504788 3491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf1bb1ab000444fbdc3e95b38eae751, State: Initialized, Role: FOLLOWER
I20260619 02:24:56.505223 3491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf1bb1ab000444fbdc3e95b38eae751" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41495 } }
I20260619 02:24:56.505367 3491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:24:56.505429 3491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:24:56.505525 3491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 02:24:56 dist-test-slave-xkrt krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:56.506376 3491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf1bb1ab000444fbdc3e95b38eae751" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41495 } }
I20260619 02:24:56.506680 3491 leader_election.cc:302] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bf1bb1ab000444fbdc3e95b38eae751; no voters:
I20260619 02:24:56.506953 3491 leader_election.cc:288] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:24:56.507058 3496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:56.507308 3496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [term 1 LEADER]: Becoming Leader. State: Replica: 6bf1bb1ab000444fbdc3e95b38eae751, State: Running, Role: LEADER
I20260619 02:24:56.507620 3496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf1bb1ab000444fbdc3e95b38eae751" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41495 } }
I20260619 02:24:56.508211 3491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:24:56.508711 3497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bf1bb1ab000444fbdc3e95b38eae751. Latest consensus state: current_term: 1 leader_uuid: "6bf1bb1ab000444fbdc3e95b38eae751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf1bb1ab000444fbdc3e95b38eae751" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41495 } } }
I20260619 02:24:56.508733 3498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bf1bb1ab000444fbdc3e95b38eae751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf1bb1ab000444fbdc3e95b38eae751" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41495 } } }
I20260619 02:24:56.508847 3497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:56.508847 3498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751 [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:56.509191 3501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:24:56.512203 3501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:24:56.512343 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:56.498920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.0.98.62:41495)
0619 02:24:56.499325 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:56.499339 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:56.499372 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:24:56.500029 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:56.500046 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:56.500336 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:56.500661 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:56.501445 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:56.502430 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:56.503304 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:56.503764 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:56.503931 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:56.506554 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:56.506559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:56.506571 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:56.506595 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:56.508585 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:56.509112 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:56.509119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:56.509125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:56.509273 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:56.509624 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:56.509628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:56.509630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:56.510167 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:56.510378 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:56.511179 (+ 801us) server_negotiation.cc:299] Negotiation successful
0619 02:24:56.511470 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260619 02:24:56.517812 3501 catalog_manager.cc:1347] Generated new cluster ID: f54589a4923c421caadaefb79414b8a0
I20260619 02:24:56.517865 3501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:24:56.533574 3501 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:24:56.534677 3501 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:24:56.543490 3501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bf1bb1ab000444fbdc3e95b38eae751: Generated new TSK 0
I20260619 02:24:56.544185 3501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:24:56.576998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41495
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:56.684665 3517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:56.684899 3517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:56.684939 3517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:56.688906 3517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:56.688959 3517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:56.689045 3517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:24:56.694151 3517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:56.695101 3517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:56.696038 3517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:56.705054 3522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:56.705194 3523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:56.705265 3517 server_base.cc:1034] running on GCE node
W20260619 02:24:56.705056 3525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:56.705760 3517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:56.706332 3517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:56.707484 3517 hybrid_clock.cc:648] HybridClock initialized: now 1781835896707474 us; error 24 us; skew 500 ppm
Jun 19 02:24:56 dist-test-slave-xkrt krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:56.710297 3517 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:24:56.711171 3517 webserver.cc:460] Webserver started at http://127.0.98.1:37017/ using document root <none> and password file <none>
I20260619 02:24:56.711685 3517 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:56.711743 3517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:56.711971 3517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:56.713901 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "30c23345fc5d408597d2f4a6b8828cb3"
format_stamp: "Formatted at 2026-06-19 02:24:56 on dist-test-slave-xkrt"
server_key: "7624038345758f75f74fcfd3b6ec222f"
server_key_iv: "10290e766eff9ce400e7742041c02b70"
server_key_version: "encryptionkey@0"
I20260619 02:24:56.714437 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "30c23345fc5d408597d2f4a6b8828cb3"
format_stamp: "Formatted at 2026-06-19 02:24:56 on dist-test-slave-xkrt"
server_key: "7624038345758f75f74fcfd3b6ec222f"
server_key_iv: "10290e766eff9ce400e7742041c02b70"
server_key_version: "encryptionkey@0"
I20260619 02:24:56.717885 3517 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:24:56.720105 3532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:56.721227 3517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:24:56.721339 3517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "30c23345fc5d408597d2f4a6b8828cb3"
format_stamp: "Formatted at 2026-06-19 02:24:56 on dist-test-slave-xkrt"
server_key: "7624038345758f75f74fcfd3b6ec222f"
server_key_iv: "10290e766eff9ce400e7742041c02b70"
server_key_version: "encryptionkey@0"
I20260619 02:24:56.721434 3517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:56.753816 3517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:56.757115 3517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:56.757319 3517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:56.757931 3517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:56.758728 3517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:56.758772 3517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:56.758836 3517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:56.758867 3517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:56.767575 3517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42703
I20260619 02:24:56.767640 3645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42703 every 8 connection(s)
I20260619 02:24:56.768453 3517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:24:56.772940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 3517
I20260619 02:24:56.773036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:24:56.773319 392 external_mini_cluster.cc:1428] Setting key 5c0e29a96f5fa55fdd65e5f99cc60805
Jun 19 02:24:56 dist-test-slave-xkrt krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:24:56.783509 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:56.770221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41181 (local address 127.0.98.62:41495)
0619 02:24:56.770350 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:56.770354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:56.771186 (+ 832us) server_negotiation.cc:407] Connection header received
0619 02:24:56.772541 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:56.772544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:56.772601 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:56.772678 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:56.774122 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:56.774823 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:56.775913 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:56.776266 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:56.776348 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:56.779442 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:56.779446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:56.779448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:56.779471 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:56.781138 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:56.781765 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:56.781768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:56.781769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:56.781879 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:56.782326 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:56.782329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:56.782330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:56.782565 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:56.782653 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:56.783168 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 02:24:56.783322 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:24:56.784230 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:56.770609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41495 (local address 127.0.98.1:41181)
0619 02:24:56.771032 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:56.771072 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:56.772333 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:56.772841 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:56.772848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:56.773285 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:56.773946 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:56.773962 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:56.774953 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:56.774955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:56.775792 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:56.775801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:56.776419 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:56.776424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:56.776637 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:56.777240 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:56.777265 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:56.779135 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:56.781257 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:56.781264 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:56.781282 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:56.781665 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:56.781999 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:56.782006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:56.782009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:56.782184 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:56.782679 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:56.782688 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:56.782953 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 02:24:56.783103 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 02:24:56.783349 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 02:24:56.785533 3646 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41495
I20260619 02:24:56.785841 3646 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:56.786314 3646 heartbeater.cc:511] Master 127.0.98.62:41495 requested a full tablet report, sending...
I20260619 02:24:56.787711 3454 ts_manager.cc:194] Registered new tserver with Master: 30c23345fc5d408597d2f4a6b8828cb3 (127.0.98.1:42703)
I20260619 02:24:56.788808 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:24:56.798887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41495
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:56.905807 3651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:56.906036 3651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:56.906076 3651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:56.910238 3651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:56.910296 3651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:56.910389 3651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:24:56.915535 3651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:56.916393 3651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:56.917290 3651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:56.926535 3657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:56.926528 3659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:56.926842 3651 server_base.cc:1034] running on GCE node
W20260619 02:24:56.926529 3656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:56.927196 3651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:56.927819 3651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:56.928977 3651 hybrid_clock.cc:648] HybridClock initialized: now 1781835896928947 us; error 41 us; skew 500 ppm
Jun 19 02:24:56 dist-test-slave-xkrt krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:56.931581 3651 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:24:56.932307 3651 webserver.cc:460] Webserver started at http://127.0.98.2:42333/ using document root <none> and password file <none>
I20260619 02:24:56.932788 3651 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:56.932843 3651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:56.932993 3651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:56.934891 3651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "dcf11fc449dc432587086b12fbf245d3"
format_stamp: "Formatted at 2026-06-19 02:24:56 on dist-test-slave-xkrt"
server_key: "de6082c486cec798ed33cca56a2d5132"
server_key_iv: "7ad30d661601db10b6c14d4f056adb51"
server_key_version: "encryptionkey@0"
I20260619 02:24:56.935357 3651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dcf11fc449dc432587086b12fbf245d3"
format_stamp: "Formatted at 2026-06-19 02:24:56 on dist-test-slave-xkrt"
server_key: "de6082c486cec798ed33cca56a2d5132"
server_key_iv: "7ad30d661601db10b6c14d4f056adb51"
server_key_version: "encryptionkey@0"
I20260619 02:24:56.938781 3651 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:24:56.940992 3666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:56.942305 3651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:24:56.942451 3651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "dcf11fc449dc432587086b12fbf245d3"
format_stamp: "Formatted at 2026-06-19 02:24:56 on dist-test-slave-xkrt"
server_key: "de6082c486cec798ed33cca56a2d5132"
server_key_iv: "7ad30d661601db10b6c14d4f056adb51"
server_key_version: "encryptionkey@0"
I20260619 02:24:56.942562 3651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:56.961669 3651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:56.964524 3651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:56.964738 3651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:56.965330 3651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:56.966223 3651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:56.966272 3651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:56.966344 3651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:56.966387 3651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:56.975696 3651 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46843
I20260619 02:24:56.975715 3779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46843 every 8 connection(s)
I20260619 02:24:56.976608 3651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:24:56.984292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 3651
I20260619 02:24:56.984452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:24:56.984735 392 external_mini_cluster.cc:1428] Setting key f44aa8eeace4edb2c719e68f40077b18
Jun 19 02:24:56 dist-test-slave-xkrt krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:56.990873 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:56.978450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46941 (local address 127.0.98.62:41495)
0619 02:24:56.978578 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:56.978582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:56.979391 (+ 809us) server_negotiation.cc:407] Connection header received
0619 02:24:56.980717 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:56.980720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:56.980766 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:56.980842 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:56.982161 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:56.982670 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:56.983880 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:56.984162 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:56.984278 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:56.987151 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:56.987157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:56.987164 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:56.987196 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:56.988673 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:56.989287 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:56.989290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:56.989292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:56.989362 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:56.989790 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:56.989792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:56.989793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:56.989955 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:56.990032 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:56.990665 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 02:24:56.990759 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:24:56.992125 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:56.978784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41495 (local address 127.0.98.2:46941)
0619 02:24:56.979209 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:56.979285 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:56.980525 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:56.981035 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:56.981042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:56.981486 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:56.982031 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:56.982045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:56.982801 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:56.982804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:56.983768 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:56.983778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:56.984309 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:56.984312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:56.984544 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:56.985195 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:56.985272 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:56.986983 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:56.988798 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:56.988806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:56.988822 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:56.989185 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:56.989449 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:56.989455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:56.989460 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:56.989692 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:56.990066 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:56.990074 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:56.990401 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 02:24:56.990609 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 02:24:56.990973 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":124,"threads_started":1}
I20260619 02:24:56.993289 3780 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41495
I20260619 02:24:56.993530 3780 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:56.994042 3780 heartbeater.cc:511] Master 127.0.98.62:41495 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:24:56.995038 3454 ts_manager.cc:194] Registered new tserver with Master: dcf11fc449dc432587086b12fbf245d3 (127.0.98.2:46843)
I20260619 02:24:56.995597 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:24:57.012241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41495
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:57.121433 3785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:57.121726 3785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:57.121795 3785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:57.126137 3785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:57.126210 3785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:57.126319 3785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:24:57.131419 3785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:57.132407 3785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:57.133384 3785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:57.142932 3793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:57.142932 3790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:57.142961 3791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:57.143146 3785 server_base.cc:1034] running on GCE node
I20260619 02:24:57.143618 3785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:57.144248 3785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:57.145439 3785 hybrid_clock.cc:648] HybridClock initialized: now 1781835897145417 us; error 52 us; skew 500 ppm
Jun 19 02:24:57 dist-test-slave-xkrt krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:57.148165 3785 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:24:57.148976 3785 webserver.cc:460] Webserver started at http://127.0.98.3:37571/ using document root <none> and password file <none>
I20260619 02:24:57.149513 3785 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:57.149654 3785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:57.149888 3785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:57.151779 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "3b4426eefe3241d581722286bc7ea6f4"
format_stamp: "Formatted at 2026-06-19 02:24:57 on dist-test-slave-xkrt"
server_key: "f4db270f44bfa8ff594444055776469a"
server_key_iv: "f5f2c0117c1e92d0f072281ba4a80729"
server_key_version: "encryptionkey@0"
I20260619 02:24:57.152325 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3b4426eefe3241d581722286bc7ea6f4"
format_stamp: "Formatted at 2026-06-19 02:24:57 on dist-test-slave-xkrt"
server_key: "f4db270f44bfa8ff594444055776469a"
server_key_iv: "f5f2c0117c1e92d0f072281ba4a80729"
server_key_version: "encryptionkey@0"
I20260619 02:24:57.155845 3785 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:24:57.158094 3800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:57.159178 3785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:24:57.159310 3785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "3b4426eefe3241d581722286bc7ea6f4"
format_stamp: "Formatted at 2026-06-19 02:24:57 on dist-test-slave-xkrt"
server_key: "f4db270f44bfa8ff594444055776469a"
server_key_iv: "f5f2c0117c1e92d0f072281ba4a80729"
server_key_version: "encryptionkey@0"
I20260619 02:24:57.159412 3785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:57.232656 3785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:57.235685 3785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:57.235906 3785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:57.236510 3785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:57.237329 3785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:57.237403 3785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:57.237475 3785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:57.237522 3785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:57.246223 3785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42307
I20260619 02:24:57.246254 3913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42307 every 8 connection(s)
I20260619 02:24:57.247207 3785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:24:57.248939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 3785
I20260619 02:24:57.249059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:24:57.249317 392 external_mini_cluster.cc:1428] Setting key def10d256e9582d5736e6e2f7d5c6cb0
Jun 19 02:24:57 dist-test-slave-xkrt krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:57.261165 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:57.248918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34173 (local address 127.0.98.62:41495)
0619 02:24:57.249100 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:57.249105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:57.250014 (+ 909us) server_negotiation.cc:407] Connection header received
0619 02:24:57.251848 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:57.251851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:57.251900 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:57.251983 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:57.253099 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:57.253801 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:57.254908 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:57.255136 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:57.255221 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:57.257689 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:57.257694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:57.257697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:57.257731 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:57.259108 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:57.259634 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:57.259640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:57.259642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:57.259739 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:57.260099 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:57.260102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:57.260104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:57.260289 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:57.260393 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:57.260888 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 02:24:57.260995 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 02:24:57.262133 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:57.249329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41495 (local address 127.0.98.3:34173)
0619 02:24:57.249824 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:57.249897 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:57.251613 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:57.252139 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:57.252146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:57.252491 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:57.252965 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:57.252978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:57.253933 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:57.253937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:57.254794 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:57.254802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:57.255263 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:57.255266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:57.255402 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:57.255998 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:57.256014 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:57.257493 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:57.259253 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:57.259258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:57.259272 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:57.259529 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:57.259857 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:57.259860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:57.259862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:57.259996 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:57.260402 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:57.260411 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:57.260646 (+ 235us) client_negotiation.cc:769] Sending connection context
0619 02:24:57.260836 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 02:24:57.261141 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":144,"threads_started":1}
I20260619 02:24:57.263283 3914 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41495
I20260619 02:24:57.263530 3914 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:57.264075 3914 heartbeater.cc:511] Master 127.0.98.62:41495 requested a full tablet report, sending...
I20260619 02:24:57.265043 3454 ts_manager.cc:194] Registered new tserver with Master: 3b4426eefe3241d581722286bc7ea6f4 (127.0.98.3:42307)
I20260619 02:24:57.265712 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34173
I20260619 02:24:57.274878 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:24:57 dist-test-slave-xkrt krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835897, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 02:24:57 dist-test-slave-xkrt krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835897, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:57.300453 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:57.291720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.0.98.62:41495)
0619 02:24:57.291895 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:57.291900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:57.292047 (+ 147us) server_negotiation.cc:407] Connection header received
0619 02:24:57.292218 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:57.292222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:57.292296 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:57.292396 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:57.293138 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:57.293787 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:57.294709 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:57.294954 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:57.295070 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:57.297675 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:57.297678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:57.297680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:57.297704 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:57.298979 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:57.299340 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:57.299342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:57.299344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:57.299419 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:57.299666 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:57.299669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:57.299670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:57.299892 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:57.300031 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:57.300241 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 02:24:57.300336 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 02:24:57.300863 3454 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60998
I20260619 02:24:57.301710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 3517
I20260619 02:24:57.307277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 3651
I20260619 02:24:57.313175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 3785
I20260619 02:24:57.318640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 3423
2026-06-19T02:24:57Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2158 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:24:57 dist-test-slave-xkrt krb5kdc[3926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:24:57 dist-test-slave-xkrt krb5kdc[3926](info): set up 2 sockets
Jun 19 02:24:57 dist-test-slave-xkrt krb5kdc[3926](info): commencing operation
krb5kdc: starting...
W20260619 02:24:58.361667 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:24:58 dist-test-slave-xkrt krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:24:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:24:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:24:58.476773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34645
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:58.585240 3939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:58.585516 3939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:58.585615 3939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:58.589412 3939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:24:58.589468 3939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:58.589488 3939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:58.589504 3939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:24:58.589517 3939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:24:58.594669 3939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:58.595569 3939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:58.596475 3939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:58.604506 3945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:58.604501 3944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:58.604501 3947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:58.604722 3939 server_base.cc:1034] running on GCE node
I20260619 02:24:58.605320 3939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:58.606379 3939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:58.607565 3939 hybrid_clock.cc:648] HybridClock initialized: now 1781835898607550 us; error 36 us; skew 500 ppm
Jun 19 02:24:58 dist-test-slave-xkrt krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:58.610144 3939 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:24:58.610994 3939 webserver.cc:460] Webserver started at http://127.0.98.62:46801/ using document root <none> and password file <none>
I20260619 02:24:58.611610 3939 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:58.611698 3939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:58.611900 3939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:58.613873 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "f63953718f6446acb9d0609b2bae9225"
format_stamp: "Formatted at 2026-06-19 02:24:58 on dist-test-slave-xkrt"
server_key: "86763c928380e7b595531743d73b4cd1"
server_key_iv: "aff199ecb6312b033c35bae698ced19f"
server_key_version: "encryptionkey@0"
I20260619 02:24:58.614413 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "f63953718f6446acb9d0609b2bae9225"
format_stamp: "Formatted at 2026-06-19 02:24:58 on dist-test-slave-xkrt"
server_key: "86763c928380e7b595531743d73b4cd1"
server_key_iv: "aff199ecb6312b033c35bae698ced19f"
server_key_version: "encryptionkey@0"
I20260619 02:24:58.617857 3939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:24:58.619999 3954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:58.621088 3939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:24:58.621217 3939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "f63953718f6446acb9d0609b2bae9225"
format_stamp: "Formatted at 2026-06-19 02:24:58 on dist-test-slave-xkrt"
server_key: "86763c928380e7b595531743d73b4cd1"
server_key_iv: "aff199ecb6312b033c35bae698ced19f"
server_key_version: "encryptionkey@0"
I20260619 02:24:58.621325 3939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:58.629494 3939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:58.632222 3939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:58.632427 3939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:58.639364 3939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34645
I20260619 02:24:58.639381 4006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34645 every 8 connection(s)
I20260619 02:24:58.640311 3939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:24:58.641886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 3939
I20260619 02:24:58.642007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:24:58.642324 392 external_mini_cluster.cc:1428] Setting key ac5c16b8a9aacd9fbf793d69fd1166fb
I20260619 02:24:58.644049 4007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:24:58.649106 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225: Bootstrap starting.
Jun 19 02:24:58 dist-test-slave-xkrt krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:58.651554 4007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225: Neither blocks nor log segments found. Creating new log.
I20260619 02:24:58.652343 4007 log.cc:824] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225: Log is configured to *not* fsync() on all Append() calls
I20260619 02:24:58.654188 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225: No bootstrap required, opened a new log
I20260619 02:24:58.656347 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:58.643699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36374 (local address 127.0.98.62:34645)
0619 02:24:58.644183 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:58.644198 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:58.644274 (+ 76us) server_negotiation.cc:407] Connection header received
0619 02:24:58.644934 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:58.644950 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:58.645272 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:58.645639 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:58.646454 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:58.647508 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:58.648414 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:58.648910 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:58.649087 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:58.651798 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:58.651804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:58.651816 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:58.651843 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:58.653697 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:58.654122 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:58.654134 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:58.654139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:58.654286 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:58.654575 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:58.654578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:58.654580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:58.654952 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:58.655157 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:58.655426 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 02:24:58.655692 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":154,"threads_started":1}
I20260619 02:24:58.657630 4007 raft_consensus.cc:348] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63953718f6446acb9d0609b2bae9225" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34645 } }
I20260619 02:24:58.657826 4007 raft_consensus.cc:374] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:24:58.657886 4007 raft_consensus.cc:729] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f63953718f6446acb9d0609b2bae9225, State: Initialized, Role: FOLLOWER
I20260619 02:24:58.658260 4007 consensus_queue.cc:260] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63953718f6446acb9d0609b2bae9225" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34645 } }
I20260619 02:24:58.658435 4007 raft_consensus.cc:388] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:24:58.658514 4007 raft_consensus.cc:482] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:24:58.658587 4007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:24:58.659310 4007 raft_consensus.cc:504] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63953718f6446acb9d0609b2bae9225" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34645 } }
I20260619 02:24:58.659582 4007 leader_election.cc:302] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f63953718f6446acb9d0609b2bae9225; no voters:
I20260619 02:24:58.659797 4007 leader_election.cc:288] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:24:58.659929 4012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:24:58.660181 4012 raft_consensus.cc:686] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [term 1 LEADER]: Becoming Leader. State: Replica: f63953718f6446acb9d0609b2bae9225, State: Running, Role: LEADER
I20260619 02:24:58.660501 4012 consensus_queue.cc:237] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63953718f6446acb9d0609b2bae9225" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34645 } }
I20260619 02:24:58.660905 4007 sys_catalog.cc:564] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:24:58.662135 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f63953718f6446acb9d0609b2bae9225. Latest consensus state: current_term: 1 leader_uuid: "f63953718f6446acb9d0609b2bae9225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63953718f6446acb9d0609b2bae9225" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34645 } } }
I20260619 02:24:58.662263 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:58.662621 4021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:24:58.662629 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f63953718f6446acb9d0609b2bae9225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63953718f6446acb9d0609b2bae9225" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34645 } } }
I20260619 02:24:58.662767 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225 [sys.catalog]: This master's current role is: LEADER
I20260619 02:24:58.665649 4021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:24:58.670879 4021 catalog_manager.cc:1347] Generated new cluster ID: 7947f09f4b6c422ab4e9546957ce8b4a
I20260619 02:24:58.670956 4021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:24:58.682631 4021 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:24:58.683503 4021 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:24:58.693858 4021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f63953718f6446acb9d0609b2bae9225: Generated new TSK 0
I20260619 02:24:58.694556 4021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:24:58.727892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34645
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:58.863065 4033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:58.863358 4033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:58.863433 4033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:58.867398 4033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:58.867484 4033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:58.867574 4033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:24:58.872723 4033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:58.873832 4033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:58.874826 4033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:58.884045 4041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:58.884205 4033 server_base.cc:1034] running on GCE node
W20260619 02:24:58.884142 4038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:58.884059 4039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:58.884763 4033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:58.885317 4033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:58.886482 4033 hybrid_clock.cc:648] HybridClock initialized: now 1781835898886455 us; error 38 us; skew 500 ppm
Jun 19 02:24:58 dist-test-slave-xkrt krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:58.889003 4033 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:24:58.889830 4033 webserver.cc:460] Webserver started at http://127.0.98.1:34987/ using document root <none> and password file <none>
I20260619 02:24:58.890317 4033 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:58.890372 4033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:58.890520 4033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:58.892392 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "0b943989bbfd43469564fb92e3c3db75"
format_stamp: "Formatted at 2026-06-19 02:24:58 on dist-test-slave-xkrt"
server_key: "2c87775f2ebf2efd733887927e7d9d92"
server_key_iv: "d2eccfd8af0da8ad055245eabd77b250"
server_key_version: "encryptionkey@0"
I20260619 02:24:58.892872 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0b943989bbfd43469564fb92e3c3db75"
format_stamp: "Formatted at 2026-06-19 02:24:58 on dist-test-slave-xkrt"
server_key: "2c87775f2ebf2efd733887927e7d9d92"
server_key_iv: "d2eccfd8af0da8ad055245eabd77b250"
server_key_version: "encryptionkey@0"
I20260619 02:24:58.896378 4033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:24:58.898710 4048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:58.900005 4033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:24:58.900137 4033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "0b943989bbfd43469564fb92e3c3db75"
format_stamp: "Formatted at 2026-06-19 02:24:58 on dist-test-slave-xkrt"
server_key: "2c87775f2ebf2efd733887927e7d9d92"
server_key_iv: "d2eccfd8af0da8ad055245eabd77b250"
server_key_version: "encryptionkey@0"
I20260619 02:24:58.900242 4033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:58.917508 4033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:58.920801 4033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:58.921002 4033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:58.921592 4033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:58.922508 4033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:58.922583 4033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:58.922676 4033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:58.922727 4033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:58.931639 4033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45521
I20260619 02:24:58.931677 4161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45521 every 8 connection(s)
I20260619 02:24:58.932495 4033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:24:58.934070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 4033
I20260619 02:24:58.934198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:24:58.934573 392 external_mini_cluster.cc:1428] Setting key 06ad5d75049504d75912adb85457b7b8
Jun 19 02:24:58 dist-test-slave-xkrt krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:24:58.947654 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:58.934318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37917 (local address 127.0.98.62:34645)
0619 02:24:58.934486 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:58.934491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:58.935513 (+ 1022us) server_negotiation.cc:407] Connection header received
0619 02:24:58.937445 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:58.937447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:58.937492 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:58.937617 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:58.938831 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:58.939432 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:58.940690 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:58.940934 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:58.941051 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:58.943951 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:58.943955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:58.943957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:58.943979 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:58.945479 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:58.946112 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:58.946115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:58.946116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:58.946185 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:58.946514 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:58.946519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:58.946521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:58.946695 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:58.946800 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:58.947357 (+ 557us) server_negotiation.cc:299] Negotiation successful
0619 02:24:58.947481 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 02:24:58.948585 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:58.934789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34645 (local address 127.0.98.1:37917)
0619 02:24:58.935371 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:58.935422 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:58.937196 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:58.937786 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:58.937799 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:58.938235 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:58.938689 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:58.938701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:58.939551 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:58.939555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:58.940568 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:58.940578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:58.941088 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:58.941092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:58.941268 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:58.941994 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:58.942010 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:58.943775 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:58.945647 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:58.945653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:58.945666 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:58.945924 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:58.946281 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:58.946285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:58.946288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:58.946421 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:58.946811 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:58.946818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:58.947116 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 02:24:58.947307 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 02:24:58.947615 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":170,"threads_started":1}
I20260619 02:24:58.949978 4162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34645
I20260619 02:24:58.950258 4162 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:58.950793 4162 heartbeater.cc:511] Master 127.0.98.62:34645 requested a full tablet report, sending...
I20260619 02:24:58.952445 3971 ts_manager.cc:194] Registered new tserver with Master: 0b943989bbfd43469564fb92e3c3db75 (127.0.98.1:45521)
I20260619 02:24:58.953634 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:24:58.965615 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34645
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:59.074472 4167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:59.074771 4167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:59.074860 4167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:59.078877 4167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:59.078949 4167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:59.079032 4167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:24:59.084550 4167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:59.085708 4167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:59.086726 4167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:59.096252 4172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:59.096228 4175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:59.096279 4167 server_base.cc:1034] running on GCE node
W20260619 02:24:59.096223 4173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:59.096994 4167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:59.097645 4167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:59.098822 4167 hybrid_clock.cc:648] HybridClock initialized: now 1781835899098807 us; error 26 us; skew 500 ppm
Jun 19 02:24:59 dist-test-slave-xkrt krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:59.101666 4167 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:24:59.102478 4167 webserver.cc:460] Webserver started at http://127.0.98.2:40701/ using document root <none> and password file <none>
I20260619 02:24:59.103071 4167 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:59.103134 4167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:59.103312 4167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:59.105865 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "8e77ff6636e34ffbbaab3ee5e813e50f"
format_stamp: "Formatted at 2026-06-19 02:24:59 on dist-test-slave-xkrt"
server_key: "fde3fcd8ded0ced83b06e673e45b058c"
server_key_iv: "50ba02d146676bd6a44b27074ed8b9d1"
server_key_version: "encryptionkey@0"
I20260619 02:24:59.106376 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8e77ff6636e34ffbbaab3ee5e813e50f"
format_stamp: "Formatted at 2026-06-19 02:24:59 on dist-test-slave-xkrt"
server_key: "fde3fcd8ded0ced83b06e673e45b058c"
server_key_iv: "50ba02d146676bd6a44b27074ed8b9d1"
server_key_version: "encryptionkey@0"
I20260619 02:24:59.109975 4167 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:24:59.112382 4182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:59.113642 4167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:24:59.113770 4167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "8e77ff6636e34ffbbaab3ee5e813e50f"
format_stamp: "Formatted at 2026-06-19 02:24:59 on dist-test-slave-xkrt"
server_key: "fde3fcd8ded0ced83b06e673e45b058c"
server_key_iv: "50ba02d146676bd6a44b27074ed8b9d1"
server_key_version: "encryptionkey@0"
I20260619 02:24:59.113868 4167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:59.127538 4167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:59.130650 4167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:59.130851 4167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:59.131410 4167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:59.132182 4167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:59.132231 4167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:59.132297 4167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:59.132336 4167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:59.141142 4167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39463
I20260619 02:24:59.141176 4295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39463 every 8 connection(s)
I20260619 02:24:59.142066 4167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:24:59 dist-test-slave-xkrt krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:59.151521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 4167
I20260619 02:24:59.151664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:24:59.151959 392 external_mini_cluster.cc:1428] Setting key d7c9d6f2f4fae4f2112ccc59ce712fa6
I20260619 02:24:59.155579 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:59.143724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44049 (local address 127.0.98.62:34645)
0619 02:24:59.143868 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:59.143871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:59.144624 (+ 753us) server_negotiation.cc:407] Connection header received
0619 02:24:59.146045 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:59.146048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:59.146094 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:59.146171 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:59.147292 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:59.147815 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:59.148928 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:59.149145 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:59.149240 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:59.151812 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:59.151816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:59.151818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:59.151841 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:59.153625 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:59.154084 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:59.154090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:59.154095 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:59.154186 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:59.154502 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:59.154508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:59.154513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:59.154753 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:59.154864 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:59.155300 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 02:24:59.155417 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:24:59.156895 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:59.144063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34645 (local address 127.0.98.2:44049)
0619 02:24:59.144477 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:59.144517 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:59.145838 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:59.146376 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:59.146383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:59.146703 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:59.147147 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:59.147159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:59.147916 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:59.147918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:59.148801 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:59.148810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:59.149314 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:59.149317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:59.149453 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:59.150002 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:59.150018 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:59.151586 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:59.153729 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:59.153733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:59.153742 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:59.153982 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:59.154284 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:59.154286 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:59.154288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:59.154411 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:59.154860 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:59.154864 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:59.155065 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 02:24:59.155250 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 02:24:59.155544 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":108,"threads_started":1}
I20260619 02:24:59.158128 4296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34645
I20260619 02:24:59.158370 4296 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:59.158824 4296 heartbeater.cc:511] Master 127.0.98.62:34645 requested a full tablet report, sending...
I20260619 02:24:59.159860 3971 ts_manager.cc:194] Registered new tserver with Master: 8e77ff6636e34ffbbaab3ee5e813e50f (127.0.98.2:39463)
I20260619 02:24:59.160645 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44049
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:24:59.178905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34645
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:24:59.286791 4301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:24:59.287019 4301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:24:59.287058 4301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:24:59.290995 4301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:24:59.291044 4301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:24:59.291121 4301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:24:59.296341 4301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:24:59.297274 4301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:24:59.298254 4301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:24:59.307500 4309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:59.307539 4306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:24:59.307547 4307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:24:59.307794 4301 server_base.cc:1034] running on GCE node
I20260619 02:24:59.308281 4301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:24:59.308889 4301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:24:59.310071 4301 hybrid_clock.cc:648] HybridClock initialized: now 1781835899310056 us; error 28 us; skew 500 ppm
Jun 19 02:24:59 dist-test-slave-xkrt krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:24:59.312729 4301 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:24:59.313619 4301 webserver.cc:460] Webserver started at http://127.0.98.3:45431/ using document root <none> and password file <none>
I20260619 02:24:59.314160 4301 fs_manager.cc:359] Metadata directory not provided
I20260619 02:24:59.314218 4301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:24:59.314373 4301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:24:59.316224 4301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "c13c0ab6e7b941ca8a48d52ec829cdb3"
format_stamp: "Formatted at 2026-06-19 02:24:59 on dist-test-slave-xkrt"
server_key: "68ede2b81056a08b64a96774b0854060"
server_key_iv: "e8d40629902706d6b5eba2d5fa535984"
server_key_version: "encryptionkey@0"
I20260619 02:24:59.316833 4301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c13c0ab6e7b941ca8a48d52ec829cdb3"
format_stamp: "Formatted at 2026-06-19 02:24:59 on dist-test-slave-xkrt"
server_key: "68ede2b81056a08b64a96774b0854060"
server_key_iv: "e8d40629902706d6b5eba2d5fa535984"
server_key_version: "encryptionkey@0"
I20260619 02:24:59.320330 4301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 02:24:59.322585 4316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:59.323887 4301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:24:59.324005 4301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "c13c0ab6e7b941ca8a48d52ec829cdb3"
format_stamp: "Formatted at 2026-06-19 02:24:59 on dist-test-slave-xkrt"
server_key: "68ede2b81056a08b64a96774b0854060"
server_key_iv: "e8d40629902706d6b5eba2d5fa535984"
server_key_version: "encryptionkey@0"
I20260619 02:24:59.324100 4301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:24:59.351745 4301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:24:59.354959 4301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:24:59.355103 4301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:24:59.355609 4301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:24:59.356333 4301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:24:59.356379 4301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:59.356416 4301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:24:59.356439 4301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:24:59.365188 4301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43723
I20260619 02:24:59.365296 4429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43723 every 8 connection(s)
I20260619 02:24:59.366115 4301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:24:59.374838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 4301
I20260619 02:24:59.374971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
Jun 19 02:24:59 dist-test-slave-xkrt krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:59.375279 392 external_mini_cluster.cc:1428] Setting key 42c7c8923a7c8aa14e834d5e9aaf6a4a
I20260619 02:24:59.379309 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:59.367812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60627 (local address 127.0.98.62:34645)
0619 02:24:59.367942 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:59.367946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:59.368722 (+ 776us) server_negotiation.cc:407] Connection header received
0619 02:24:59.370098 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:59.370101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:59.370145 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:59.370220 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:59.371299 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:59.371807 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:59.372877 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:59.373087 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:59.373162 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:59.375738 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:59.375745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:59.375750 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:59.375778 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:59.377244 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:59.377814 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:59.377819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:59.377821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:59.377905 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:59.378244 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:59.378247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:59.378249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:59.378421 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:59.378531 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:59.379029 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 02:24:59.379147 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:24:59.380045 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:59.368162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34645 (local address 127.0.98.3:60627)
0619 02:24:59.368567 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 02:24:59.368607 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:24:59.369906 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:24:59.370372 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:24:59.370378 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:24:59.370719 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:24:59.371176 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:59.371188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:59.371914 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:59.371917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:59.372775 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:24:59.372783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:24:59.373206 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:24:59.373209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:24:59.373340 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:59.373864 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:24:59.373879 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:24:59.375555 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:24:59.377364 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:59.377370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:59.377383 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:59.377715 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:59.378002 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:24:59.378005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:24:59.378008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:24:59.378146 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:24:59.378540 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:24:59.378549 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:24:59.378781 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 02:24:59.378974 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 02:24:59.379271 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":108,"threads_started":1}
I20260619 02:24:59.381091 4430 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34645
I20260619 02:24:59.381322 4430 heartbeater.cc:464] Registering TS with master...
I20260619 02:24:59.381850 4430 heartbeater.cc:511] Master 127.0.98.62:34645 requested a full tablet report, sending...
I20260619 02:24:59.382772 3971 ts_manager.cc:194] Registered new tserver with Master: c13c0ab6e7b941ca8a48d52ec829cdb3 (127.0.98.3:43723)
I20260619 02:24:59.383348 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60627
I20260619 02:24:59.388741 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:24:59 dist-test-slave-xkrt krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:24:59 dist-test-slave-xkrt krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 02:24:59.413246 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:59.404317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.0.98.1:45521)
0619 02:24:59.404589 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:59.404604 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:59.404630 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:24:59.404774 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:59.404780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:59.404975 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:59.405128 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:59.405896 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:59.406649 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:59.407429 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:59.407779 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:59.407878 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:59.410281 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:59.410288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:59.410300 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:59.410321 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:59.411726 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:59.412138 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:59.412142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:59.412144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:59.412216 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:59.412448 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:59.412450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:59.412451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:59.412660 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:59.412823 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:59.412988 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 02:24:59.413101 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":63,"threads_started":1}
Jun 19 02:24:59 dist-test-slave-xkrt krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:24:59.423887 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:24:59.416013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36384 (local address 127.0.98.62:34645)
0619 02:24:59.416244 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 02:24:59.416252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:24:59.416287 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:24:59.416419 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:24:59.416423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:24:59.416475 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:24:59.416561 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:24:59.417223 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:59.417822 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:59.418658 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:24:59.418871 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:24:59.418976 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:24:59.421185 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:24:59.421188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:24:59.421190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:24:59.421210 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:24:59.422396 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:59.422758 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:59.422761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:59.422762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:59.422827 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:24:59.423047 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:24:59.423049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:24:59.423050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:24:59.423243 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:24:59.423391 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 02:24:59.423624 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 02:24:59.423733 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
W20260619 02:24:59.424301 3971 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36384
I20260619 02:24:59.424907 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 4033
I20260619 02:24:59.430441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 4167
I20260619 02:24:59.435734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 4301
I20260619 02:24:59.440754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 3939
2026-06-19T02:24:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2121 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:24:59 dist-test-slave-xkrt krb5kdc[4442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:24:59 dist-test-slave-xkrt krb5kdc[4442](info): set up 2 sockets
Jun 19 02:24:59 dist-test-slave-xkrt krb5kdc[4442](info): commencing operation
krb5kdc: starting...
W20260619 02:25:00.485666 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:25:00 dist-test-slave-xkrt krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:25:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:25:00.602560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43137
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:00.706730 4455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:00.707017 4455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:00.707093 4455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:00.711051 4455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:00.711134 4455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:00.711172 4455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:00.711228 4455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:00.711267 4455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:25:00.716289 4455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:00.717259 4455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:00.718240 4455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:00.726280 4461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:00.726347 4463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:00.726310 4455 server_base.cc:1034] running on GCE node
W20260619 02:25:00.726495 4460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:00.726985 4455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:00.728046 4455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:00.729233 4455 hybrid_clock.cc:648] HybridClock initialized: now 1781835900729221 us; error 26 us; skew 500 ppm
Jun 19 02:25:00 dist-test-slave-xkrt krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:00.735217 4455 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:25:00.735944 4455 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:00.736034 4455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:00.736251 4455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:00.738255 4455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "bdc73a1aa96d434a95255c292df1a763"
format_stamp: "Formatted at 2026-06-19 02:25:00 on dist-test-slave-xkrt"
server_key: "2fff64639918fc470365a2346f254f07"
server_key_iv: "7eaca8dfc2eaef0118a6dfbddbdc783b"
server_key_version: "encryptionkey@0"
I20260619 02:25:00.738791 4455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "bdc73a1aa96d434a95255c292df1a763"
format_stamp: "Formatted at 2026-06-19 02:25:00 on dist-test-slave-xkrt"
server_key: "2fff64639918fc470365a2346f254f07"
server_key_iv: "7eaca8dfc2eaef0118a6dfbddbdc783b"
server_key_version: "encryptionkey@0"
I20260619 02:25:00.742233 4455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:25:00.744464 4469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:00.745689 4455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:25:00.745815 4455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "bdc73a1aa96d434a95255c292df1a763"
format_stamp: "Formatted at 2026-06-19 02:25:00 on dist-test-slave-xkrt"
server_key: "2fff64639918fc470365a2346f254f07"
server_key_iv: "7eaca8dfc2eaef0118a6dfbddbdc783b"
server_key_version: "encryptionkey@0"
I20260619 02:25:00.745918 4455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:00.768332 4455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:00.771198 4455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:00.771396 4455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:00.778232 4455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43137
I20260619 02:25:00.778514 4521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43137 every 8 connection(s)
I20260619 02:25:00.778964 4455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:25:00.781647 4522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:00.786924 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763: Bootstrap starting.
I20260619 02:25:00.787931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 4455
I20260619 02:25:00.788035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:25:00.788301 392 external_mini_cluster.cc:1428] Setting key 05d54e49b332d66d294f881e450f652d
I20260619 02:25:00.789705 4522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:00.790557 4522 log.cc:824] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:00.792572 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763: No bootstrap required, opened a new log
I20260619 02:25:00.796450 4522 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc73a1aa96d434a95255c292df1a763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43137 } }
I20260619 02:25:00.796746 4522 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:00.796813 4522 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdc73a1aa96d434a95255c292df1a763, State: Initialized, Role: FOLLOWER
Jun 19 02:25:00 dist-test-slave-xkrt krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:25:00.797302 4522 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc73a1aa96d434a95255c292df1a763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43137 } }
I20260619 02:25:00.797477 4522 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:25:00.797571 4522 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:25:00.797675 4522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:00.798980 4522 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc73a1aa96d434a95255c292df1a763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43137 } }
I20260619 02:25:00.799320 4522 leader_election.cc:302] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdc73a1aa96d434a95255c292df1a763; no voters:
I20260619 02:25:00.799633 4522 leader_election.cc:288] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:25:00.799739 4527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:00.799945 4527 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [term 1 LEADER]: Becoming Leader. State: Replica: bdc73a1aa96d434a95255c292df1a763, State: Running, Role: LEADER
I20260619 02:25:00.800323 4527 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc73a1aa96d434a95255c292df1a763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43137 } }
I20260619 02:25:00.800944 4522 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:00.802011 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdc73a1aa96d434a95255c292df1a763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc73a1aa96d434a95255c292df1a763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43137 } } }
I20260619 02:25:00.802134 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:00.802537 4529 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdc73a1aa96d434a95255c292df1a763. Latest consensus state: current_term: 1 leader_uuid: "bdc73a1aa96d434a95255c292df1a763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc73a1aa96d434a95255c292df1a763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43137 } } }
I20260619 02:25:00.802673 4529 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:00.803073 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:00.789393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.0.98.62:43137)
0619 02:25:00.789874 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:00.789899 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:00.789947 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:25:00.790726 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:00.790748 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:00.791120 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:00.791472 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:00.792093 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:00.793508 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:00.794331 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:00.795032 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:00.795191 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:00.797809 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:00.797815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:00.797836 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:00.797862 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:00.799670 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:00.800208 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:00.800213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:00.800219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:00.800352 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:00.800670 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:00.800673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:00.800674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:00.801151 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:00.801337 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:00.801700 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 02:25:00.802038 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":151,"threads_started":1}
W20260619 02:25:00.805523 4542 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:25:00.805684 4542 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:25:00.805785 4536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:00.806553 4536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:00.811774 4536 catalog_manager.cc:1347] Generated new cluster ID: 24a68242e2b148d496a8bc89fb70b3ff
I20260619 02:25:00.811836 4536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:00.838816 4536 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:25:00.839854 4536 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:00.845566 4536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763: Generated new TSK 0
I20260619 02:25:00.846331 4536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:25:00.874468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43137
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:00.980407 4548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:00.980633 4548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:00.980674 4548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:00.984596 4548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:00.984649 4548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:00.984735 4548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:25:00.989883 4548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:00.990849 4548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:00.991819 4548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:01.000905 4553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:01.000982 4548 server_base.cc:1034] running on GCE node
W20260619 02:25:01.000977 4554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:01.000911 4556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:01.001652 4548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:01.002197 4548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:01.003379 4548 hybrid_clock.cc:648] HybridClock initialized: now 1781835901003344 us; error 55 us; skew 500 ppm
Jun 19 02:25:01 dist-test-slave-xkrt krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:01.006189 4548 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:25:01.006948 4548 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:01.007015 4548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:01.007216 4548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:01.009159 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "21431b6d1a86497a8d90dad419436f0c"
format_stamp: "Formatted at 2026-06-19 02:25:01 on dist-test-slave-xkrt"
server_key: "520d00dcab9a1beee9683337c08b24ea"
server_key_iv: "620434dcc4fff5e30986779b57dd1da7"
server_key_version: "encryptionkey@0"
I20260619 02:25:01.009802 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "21431b6d1a86497a8d90dad419436f0c"
format_stamp: "Formatted at 2026-06-19 02:25:01 on dist-test-slave-xkrt"
server_key: "520d00dcab9a1beee9683337c08b24ea"
server_key_iv: "620434dcc4fff5e30986779b57dd1da7"
server_key_version: "encryptionkey@0"
I20260619 02:25:01.013259 4548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:25:01.015607 4562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:01.016737 4548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:01.016867 4548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "21431b6d1a86497a8d90dad419436f0c"
format_stamp: "Formatted at 2026-06-19 02:25:01 on dist-test-slave-xkrt"
server_key: "520d00dcab9a1beee9683337c08b24ea"
server_key_iv: "620434dcc4fff5e30986779b57dd1da7"
server_key_version: "encryptionkey@0"
I20260619 02:25:01.016979 4548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:01.043277 4548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:01.046188 4548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:01.046391 4548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:01.046980 4548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:01.047786 4548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:01.047857 4548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:01.047928 4548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:01.047991 4548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:01.056910 4548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34155
I20260619 02:25:01.056932 4675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34155 every 8 connection(s)
I20260619 02:25:01.057480 4548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:25:01.060158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 4548
I20260619 02:25:01.060289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:25:01.060580 392 external_mini_cluster.cc:1428] Setting key 78272af681b031c4c342191deaa10ec0
Jun 19 02:25:01 dist-test-slave-xkrt krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 02:25:01.071599 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.059190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37297 (local address 127.0.98.62:43137)
0619 02:25:01.059324 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:01.059328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:01.060177 (+ 849us) server_negotiation.cc:407] Connection header received
0619 02:25:01.061801 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:01.061806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:01.061860 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:01.061941 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:01.063094 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.063858 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.064991 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.065262 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.065373 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.067818 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:01.067822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:01.067824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:01.067856 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:01.069371 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:01.069902 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:01.069907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:01.069908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:01.069988 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:01.070411 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:01.070414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:01.070416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:01.070586 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:01.070692 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:01.071280 (+ 588us) server_negotiation.cc:299] Negotiation successful
0619 02:25:01.071415 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:25:01.072588 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.059577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43137 (local address 127.0.98.1:37297)
0619 02:25:01.060037 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:01.060079 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:01.061522 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:01.062138 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:01.062145 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:01.062496 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:01.062951 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.062963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.063985 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.063989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.064869 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.064878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.065407 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.065411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.065649 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.066257 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:01.066272 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:01.067649 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:01.069482 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:01.069488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:01.069501 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:01.069798 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:01.070115 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:01.070120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:01.070125 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:01.070305 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:01.070690 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:01.070697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:01.071038 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 02:25:01.071220 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 02:25:01.071572 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":119,"threads_started":1}
I20260619 02:25:01.073944 4676 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43137
I20260619 02:25:01.074186 4676 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:01.074640 4676 heartbeater.cc:511] Master 127.0.98.62:43137 requested a full tablet report, sending...
I20260619 02:25:01.076231 4486 ts_manager.cc:194] Registered new tserver with Master: 21431b6d1a86497a8d90dad419436f0c (127.0.98.1:34155)
I20260619 02:25:01.077378 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:25:01.089850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43137
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:01.196141 4681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:01.196401 4681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:01.196445 4681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:01.200496 4681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:01.200555 4681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:01.200614 4681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:25:01.205996 4681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:01.206991 4681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:01.207974 4681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:01.217181 4689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:01.217351 4681 server_base.cc:1034] running on GCE node
W20260619 02:25:01.217239 4686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:01.217307 4687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:01.217885 4681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:01.218508 4681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:01.219662 4681 hybrid_clock.cc:648] HybridClock initialized: now 1781835901219651 us; error 44 us; skew 500 ppm
Jun 19 02:25:01 dist-test-slave-xkrt krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:01.222685 4681 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:25:01.223484 4681 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:01.223548 4681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:01.223750 4681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:01.225791 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "90bce0a16c23458b9afdead97aa131cf"
format_stamp: "Formatted at 2026-06-19 02:25:01 on dist-test-slave-xkrt"
server_key: "136bb41eb2f05b84a678b2adb8b125cb"
server_key_iv: "9c38ee0d6a2578f3155dbe7b9791fdff"
server_key_version: "encryptionkey@0"
I20260619 02:25:01.226333 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "90bce0a16c23458b9afdead97aa131cf"
format_stamp: "Formatted at 2026-06-19 02:25:01 on dist-test-slave-xkrt"
server_key: "136bb41eb2f05b84a678b2adb8b125cb"
server_key_iv: "9c38ee0d6a2578f3155dbe7b9791fdff"
server_key_version: "encryptionkey@0"
I20260619 02:25:01.229956 4681 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:25:01.232183 4695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:01.233413 4681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:01.233588 4681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "90bce0a16c23458b9afdead97aa131cf"
format_stamp: "Formatted at 2026-06-19 02:25:01 on dist-test-slave-xkrt"
server_key: "136bb41eb2f05b84a678b2adb8b125cb"
server_key_iv: "9c38ee0d6a2578f3155dbe7b9791fdff"
server_key_version: "encryptionkey@0"
I20260619 02:25:01.233700 4681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:01.249784 4681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:01.252655 4681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:01.252846 4681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:01.253384 4681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:01.254222 4681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:01.254292 4681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:01.254356 4681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:01.254405 4681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:01.262977 4681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36843
I20260619 02:25:01.263016 4808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36843 every 8 connection(s)
I20260619 02:25:01.263540 4681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:25:01.265456 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 4681
I20260619 02:25:01.265607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:25:01.265882 392 external_mini_cluster.cc:1428] Setting key 39419e3498da71ae8c529887929b0fe1
Jun 19 02:25:01 dist-test-slave-xkrt krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 02:25:01.278895 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.265127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46679 (local address 127.0.98.62:43137)
0619 02:25:01.265252 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:01.265256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:01.266130 (+ 874us) server_negotiation.cc:407] Connection header received
0619 02:25:01.267624 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:01.267631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:01.267693 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:01.267787 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:01.269087 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.269674 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.271471 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.271679 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.271757 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.274607 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:01.274611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:01.274613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:01.274640 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:01.276404 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:01.277106 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:01.277111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:01.277114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:01.277255 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:01.277704 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:01.277706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:01.277707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:01.277961 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:01.278067 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:01.278612 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 02:25:01.278731 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:25:01.279714 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.265438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43137 (local address 127.0.98.2:46679)
0619 02:25:01.265963 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:01.266013 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:01.267414 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:01.267929 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:01.267936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:01.268317 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:01.268927 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.268946 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.269794 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.269801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.271334 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.271348 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.271825 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.271832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.272064 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.272665 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:01.272691 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:01.274349 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:01.276563 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:01.276571 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:01.276586 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:01.276993 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:01.277373 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:01.277379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:01.277382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:01.277630 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:01.278093 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:01.278101 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:01.278368 (+ 267us) client_negotiation.cc:769] Sending connection context
0619 02:25:01.278556 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 02:25:01.278896 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":154,"threads_started":1}
I20260619 02:25:01.280928 4809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43137
I20260619 02:25:01.281190 4809 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:01.281728 4809 heartbeater.cc:511] Master 127.0.98.62:43137 requested a full tablet report, sending...
I20260619 02:25:01.282814 4486 ts_manager.cc:194] Registered new tserver with Master: 90bce0a16c23458b9afdead97aa131cf (127.0.98.2:36843)
I20260619 02:25:01.283748 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:25:01.294634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43137
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:01.407650 4814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:01.407909 4814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:01.407954 4814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:01.411995 4814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:01.412053 4814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:01.412142 4814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:25:01.417359 4814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:01.418428 4814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:01.419427 4814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:01.428889 4819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:01.428922 4820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:01.428889 4822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:01.428963 4814 server_base.cc:1034] running on GCE node
I20260619 02:25:01.429673 4814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:01.430348 4814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:01.431528 4814 hybrid_clock.cc:648] HybridClock initialized: now 1781835901431515 us; error 27 us; skew 500 ppm
Jun 19 02:25:01 dist-test-slave-xkrt krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:01.434335 4814 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:25:01.435047 4814 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:01.435142 4814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:01.435366 4814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:01.437528 4814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "b0437c9edd764e29bc6fc894aba9f4c9"
format_stamp: "Formatted at 2026-06-19 02:25:01 on dist-test-slave-xkrt"
server_key: "6f6af5079205194bd1d1023b6c43bd48"
server_key_iv: "14139eba7149f7ad1670ed4844e43da3"
server_key_version: "encryptionkey@0"
I20260619 02:25:01.438127 4814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b0437c9edd764e29bc6fc894aba9f4c9"
format_stamp: "Formatted at 2026-06-19 02:25:01 on dist-test-slave-xkrt"
server_key: "6f6af5079205194bd1d1023b6c43bd48"
server_key_iv: "14139eba7149f7ad1670ed4844e43da3"
server_key_version: "encryptionkey@0"
I20260619 02:25:01.441885 4814 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 02:25:01.444206 4828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:01.445500 4814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:01.445680 4814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "b0437c9edd764e29bc6fc894aba9f4c9"
format_stamp: "Formatted at 2026-06-19 02:25:01 on dist-test-slave-xkrt"
server_key: "6f6af5079205194bd1d1023b6c43bd48"
server_key_iv: "14139eba7149f7ad1670ed4844e43da3"
server_key_version: "encryptionkey@0"
I20260619 02:25:01.445791 4814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:01.460507 4814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:01.463338 4814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:01.463537 4814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:01.464097 4814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:01.464900 4814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:01.464952 4814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:01.464991 4814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:01.465047 4814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:01.473981 4814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40399
I20260619 02:25:01.474047 4941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40399 every 8 connection(s)
I20260619 02:25:01.474615 4814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:25:01.480821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 4814
I20260619 02:25:01.480938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:25:01.481217 392 external_mini_cluster.cc:1428] Setting key 4540df2db82f3361fbfb281146699762
Jun 19 02:25:01 dist-test-slave-xkrt krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:25:01.488385 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.476353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56599 (local address 127.0.98.62:43137)
0619 02:25:01.476503 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:01.476506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:01.477339 (+ 833us) server_negotiation.cc:407] Connection header received
0619 02:25:01.478726 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:01.478729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:01.478788 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:01.478866 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:01.480074 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.480642 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.482156 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.482471 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.482589 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.485147 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:01.485150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:01.485152 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:01.485177 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:01.486501 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:01.487060 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:01.487062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:01.487064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:01.487136 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:01.487453 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:01.487455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:01.487456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:01.487623 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:01.487696 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:01.488175 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 02:25:01.488270 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:25:01.489187 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.476720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43137 (local address 127.0.98.3:56599)
0619 02:25:01.477169 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:01.477210 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:01.478521 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:01.479031 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:01.479038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:01.479418 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:01.479884 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.479898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.480774 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.480778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.482029 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.482041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.482623 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.482627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.482794 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.483448 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:01.483464 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:01.484973 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:01.486652 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:01.486657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:01.486669 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:01.486969 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:01.487215 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:01.487217 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:01.487219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:01.487361 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:01.487732 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:01.487737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:01.487972 (+ 235us) client_negotiation.cc:769] Sending connection context
0619 02:25:01.488132 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 02:25:01.488378 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":110,"threads_started":1}
I20260619 02:25:01.490351 4942 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43137
I20260619 02:25:01.490592 4942 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:01.491043 4942 heartbeater.cc:511] Master 127.0.98.62:43137 requested a full tablet report, sending...
I20260619 02:25:01.492022 4486 ts_manager.cc:194] Registered new tserver with Master: b0437c9edd764e29bc6fc894aba9f4c9 (127.0.98.3:40399)
I20260619 02:25:01.492542 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56599
I20260619 02:25:01.494949 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:25:01.504102 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.496808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.0.98.62:43137)
0619 02:25:01.496946 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:01.496950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:01.497084 (+ 134us) server_negotiation.cc:407] Connection header received
0619 02:25:01.497165 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:01.497167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:01.497203 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:01.497266 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:01.497944 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.498458 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.499310 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.499539 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.499614 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.500986 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:01.500988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:01.500991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:01.501013 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:01.502554 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:01.502953 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:01.502955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:01.502957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:01.503030 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:01.503327 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:01.503330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:01.503331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:01.503533 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:01.503645 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:01.503805 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 02:25:01.503901 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:25:01.506953 4486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:25:01.510010 4486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:25:01.524681 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.518210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40399 (local address 127.0.0.1:37606)
0619 02:25:01.518641 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:01.518688 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:01.518840 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:01.520155 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:01.520164 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:01.520193 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:01.520334 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.520351 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.521841 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.521845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.523064 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.523074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.524111 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.524115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.524224 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.524266 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 02:25:01.524387 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 02:25:01.524510 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":20,"thread_start_us":151,"threads_started":1}
I20260619 02:25:01.524696 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.519341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37606 (local address 127.0.98.3:40399)
0619 02:25:01.519645 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:01.519659 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:01.519684 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:25:01.519765 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:01.519774 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:01.519993 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:01.520153 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:01.520478 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.521693 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.523190 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.523984 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.524092 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.524228 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:01.524422 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 02:25:01.524511 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":129,"threads_started":1}
I20260619 02:25:01.525377 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.519092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.0.98.2:36843)
0619 02:25:01.519625 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:01.519640 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:01.519808 (+ 168us) server_negotiation.cc:407] Connection header received
0619 02:25:01.519931 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:01.519935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:01.520103 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:01.520288 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:01.520884 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.521983 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.523712 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.524363 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.524463 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.524586 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:01.525129 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 02:25:01.525223 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":117,"threads_started":1}
I20260619 02:25:01.526008 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.518310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34155 (local address 127.0.0.1:47342)
0619 02:25:01.519278 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:01.519293 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:01.519388 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:01.520516 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:01.520519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:01.520549 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:01.520760 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.520764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.523456 (+ 2692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.523459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.524528 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.524538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.525652 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.525655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.525758 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.525787 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 02:25:01.525847 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:25:01.525914 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":75,"spinlock_wait_cycles":8320,"thread_start_us":96,"threads_started":1}
I20260619 02:25:01.525035 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.518949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36843 (local address 127.0.0.1:59462)
0619 02:25:01.519726 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:01.519739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:01.519830 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:01.520516 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:01.520519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:01.520544 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:01.520758 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.520764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.522105 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.522111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.523350 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.523360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.524645 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.524648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.524746 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.524772 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 02:25:01.524853 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:25:01.524916 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":70,"spinlock_wait_cycles":9344,"thread_start_us":85,"threads_started":1}
I20260619 02:25:01.526854 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.518909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.0.98.1:34155)
0619 02:25:01.520057 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:01.520065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:01.520090 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:25:01.520160 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:01.520164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:01.520363 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:01.520539 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:01.522149 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.523323 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.524662 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.525435 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.526453 (+ 1018us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.526582 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:01.526631 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 02:25:01.526709 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1048,"thread_start_us":113,"threads_started":1}
I20260619 02:25:01.528225 4876 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8f068eb3204214815ba74b4c586e14 (DEFAULT_TABLE table=test-table [id=3764cd320c3c49679412b6dd70e5a42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:01.528659 4743 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8f068eb3204214815ba74b4c586e14 (DEFAULT_TABLE table=test-table [id=3764cd320c3c49679412b6dd70e5a42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:01.529299 4876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8f068eb3204214815ba74b4c586e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:01.529696 4743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8f068eb3204214815ba74b4c586e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:01.529325 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8f068eb3204214815ba74b4c586e14 (DEFAULT_TABLE table=test-table [id=3764cd320c3c49679412b6dd70e5a42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:01.530308 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8f068eb3204214815ba74b4c586e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:01.535229 4959 tablet_bootstrap.cc:492] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9: Bootstrap starting.
I20260619 02:25:01.536649 4960 tablet_bootstrap.cc:492] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf: Bootstrap starting.
I20260619 02:25:01.537173 4961 tablet_bootstrap.cc:492] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c: Bootstrap starting.
I20260619 02:25:01.538496 4959 tablet_bootstrap.cc:654] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:01.538784 4960 tablet_bootstrap.cc:654] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:01.539186 4961 tablet_bootstrap.cc:654] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:01.539927 4959 log.cc:824] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:01.539927 4960 log.cc:824] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:01.540220 4961 log.cc:824] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:01.542024 4961 tablet_bootstrap.cc:492] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c: No bootstrap required, opened a new log
I20260619 02:25:01.542098 4960 tablet_bootstrap.cc:492] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf: No bootstrap required, opened a new log
I20260619 02:25:01.542215 4959 tablet_bootstrap.cc:492] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9: No bootstrap required, opened a new log
I20260619 02:25:01.542274 4961 ts_tablet_manager.cc:1397] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:25:01.542326 4960 ts_tablet_manager.cc:1397] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:25:01.542375 4959 ts_tablet_manager.cc:1397] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 02:25:01.545629 4961 raft_consensus.cc:348] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } } peers { permanent_uuid: "21431b6d1a86497a8d90dad419436f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "90bce0a16c23458b9afdead97aa131cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260619 02:25:01.545878 4961 raft_consensus.cc:374] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:01.545949 4961 raft_consensus.cc:729] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21431b6d1a86497a8d90dad419436f0c, State: Initialized, Role: FOLLOWER
I20260619 02:25:01.546240 4960 raft_consensus.cc:348] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } } peers { permanent_uuid: "21431b6d1a86497a8d90dad419436f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "90bce0a16c23458b9afdead97aa131cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260619 02:25:01.546240 4959 raft_consensus.cc:348] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } } peers { permanent_uuid: "21431b6d1a86497a8d90dad419436f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "90bce0a16c23458b9afdead97aa131cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260619 02:25:01.546506 4960 raft_consensus.cc:374] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:01.546424 4961 consensus_queue.cc:260] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } } peers { permanent_uuid: "21431b6d1a86497a8d90dad419436f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "90bce0a16c23458b9afdead97aa131cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260619 02:25:01.546535 4959 raft_consensus.cc:374] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:01.546587 4959 raft_consensus.cc:729] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0437c9edd764e29bc6fc894aba9f4c9, State: Initialized, Role: FOLLOWER
I20260619 02:25:01.546613 4960 raft_consensus.cc:729] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90bce0a16c23458b9afdead97aa131cf, State: Initialized, Role: FOLLOWER
I20260619 02:25:01.547036 4960 consensus_queue.cc:260] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } } peers { permanent_uuid: "21431b6d1a86497a8d90dad419436f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "90bce0a16c23458b9afdead97aa131cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260619 02:25:01.547044 4959 consensus_queue.cc:260] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } } peers { permanent_uuid: "21431b6d1a86497a8d90dad419436f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "90bce0a16c23458b9afdead97aa131cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260619 02:25:01.547111 4676 heartbeater.cc:503] Master 127.0.98.62:43137 was elected leader, sending a full tablet report...
I20260619 02:25:01.547477 4961 ts_tablet_manager.cc:1428] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:25:01.547968 4960 ts_tablet_manager.cc:1428] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260619 02:25:01.547968 4959 ts_tablet_manager.cc:1428] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:25:01.548286 4809 heartbeater.cc:503] Master 127.0.98.62:43137 was elected leader, sending a full tablet report...
I20260619 02:25:01.548316 4942 heartbeater.cc:503] Master 127.0.98.62:43137 was elected leader, sending a full tablet report...
W20260619 02:25:01.558490 4677 tablet.cc:2369] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:25:01.697868 4966 raft_consensus.cc:482] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:25:01.698114 4966 raft_consensus.cc:504] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } } peers { permanent_uuid: "21431b6d1a86497a8d90dad419436f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "90bce0a16c23458b9afdead97aa131cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260619 02:25:01.699373 4966 leader_election.cc:288] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21431b6d1a86497a8d90dad419436f0c (127.0.98.1:34155), 90bce0a16c23458b9afdead97aa131cf (127.0.98.2:36843)
I20260619 02:25:01.703378 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.699743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45559 (local address 127.0.98.1:34155)
0619 02:25:01.699876 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:01.699883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:01.699920 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:25:01.700123 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:01.700128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:01.700184 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:01.700294 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:01.700565 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.701138 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.702423 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.702808 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.702931 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.703006 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:01.703198 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 02:25:01.703278 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:25:01.703372 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.699678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34155 (local address 127.0.98.3:45559)
0619 02:25:01.699836 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:01.699853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:01.699955 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:01.700303 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:01.700306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:01.700323 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:01.700438 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.700459 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.701251 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.701255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.702281 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.702290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.702932 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.702935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.703039 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.703060 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:25:01.703152 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:25:01.703238 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":19}
I20260619 02:25:01.703902 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.699967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36843 (local address 127.0.98.3:34391)
0619 02:25:01.700328 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:01.700340 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:01.700459 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:01.700688 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:01.700691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:01.700705 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:01.700822 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.700833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.701655 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.701660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.702667 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:01.702676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.703444 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.703454 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:01.703558 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.703575 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:25:01.703695 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 02:25:01.703759 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":1,"thread_start_us":132,"threads_started":1}
I20260619 02:25:01.703917 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.700117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34391 (local address 127.0.98.2:36843)
0619 02:25:01.700279 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:01.700284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:01.700421 (+ 137us) server_negotiation.cc:407] Connection header received
0619 02:25:01.700542 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:01.700546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:01.700597 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:01.700678 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:01.700930 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.701505 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.702786 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.703311 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.703431 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.703489 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:01.703725 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 02:25:01.703809 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 02:25:01.704054 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8f068eb3204214815ba74b4c586e14" candidate_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21431b6d1a86497a8d90dad419436f0c" is_pre_election: true
I20260619 02:25:01.704380 4630 raft_consensus.cc:2454] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0437c9edd764e29bc6fc894aba9f4c9 in term 0.
I20260619 02:25:01.704911 4829 leader_election.cc:302] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21431b6d1a86497a8d90dad419436f0c, b0437c9edd764e29bc6fc894aba9f4c9; no voters:
I20260619 02:25:01.705207 4966 raft_consensus.cc:2781] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:25:01.705132 4763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8f068eb3204214815ba74b4c586e14" candidate_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90bce0a16c23458b9afdead97aa131cf" is_pre_election: true
I20260619 02:25:01.705303 4966 raft_consensus.cc:482] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:25:01.705350 4966 raft_consensus.cc:3037] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:01.705467 4763 raft_consensus.cc:2454] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0437c9edd764e29bc6fc894aba9f4c9 in term 0.
I20260619 02:25:01.706462 4966 raft_consensus.cc:504] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } } peers { permanent_uuid: "21431b6d1a86497a8d90dad419436f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "90bce0a16c23458b9afdead97aa131cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260619 02:25:01.706753 4966 leader_election.cc:288] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [CANDIDATE]: Term 1 election: Requested vote from peers 21431b6d1a86497a8d90dad419436f0c (127.0.98.1:34155), 90bce0a16c23458b9afdead97aa131cf (127.0.98.2:36843)
I20260619 02:25:01.707100 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8f068eb3204214815ba74b4c586e14" candidate_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21431b6d1a86497a8d90dad419436f0c"
I20260619 02:25:01.707242 4630 raft_consensus.cc:3037] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:01.707229 4763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8f068eb3204214815ba74b4c586e14" candidate_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90bce0a16c23458b9afdead97aa131cf"
I20260619 02:25:01.707360 4763 raft_consensus.cc:3037] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:01.708375 4630 raft_consensus.cc:2454] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0437c9edd764e29bc6fc894aba9f4c9 in term 1.
I20260619 02:25:01.708592 4763 raft_consensus.cc:2454] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0437c9edd764e29bc6fc894aba9f4c9 in term 1.
I20260619 02:25:01.708714 4829 leader_election.cc:302] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21431b6d1a86497a8d90dad419436f0c, b0437c9edd764e29bc6fc894aba9f4c9; no voters:
I20260619 02:25:01.708925 4966 raft_consensus.cc:2781] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:01.709183 4966 raft_consensus.cc:686] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [term 1 LEADER]: Becoming Leader. State: Replica: b0437c9edd764e29bc6fc894aba9f4c9, State: Running, Role: LEADER
I20260619 02:25:01.709510 4966 consensus_queue.cc:237] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } } peers { permanent_uuid: "21431b6d1a86497a8d90dad419436f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "90bce0a16c23458b9afdead97aa131cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260619 02:25:01.712761 4485 catalog_manager.cc:5495] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0437c9edd764e29bc6fc894aba9f4c9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0437c9edd764e29bc6fc894aba9f4c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21431b6d1a86497a8d90dad419436f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90bce0a16c23458b9afdead97aa131cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } health_report { overall_health: UNKNOWN } } }
W20260619 02:25:01.725693 4943 tablet.cc:2369] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:25:01.763175 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.759662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37616 (local address 127.0.98.3:40399)
0619 02:25:01.759825 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:01.759829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:01.759844 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:25:01.759923 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:01.759925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:01.759990 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:01.760081 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:25:01.760359 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.760843 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.761718 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:01.761916 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:01.762001 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:01.762262 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:25:01.762743 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:25:01.762839 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:01.762922 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 02:25:01.762980 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 02:25:01.764554 4810 tablet.cc:2369] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:25:01.769037 4630 raft_consensus.cc:1261] T 1c8f068eb3204214815ba74b4c586e14 P 21431b6d1a86497a8d90dad419436f0c [term 1 FOLLOWER]: Refusing update from remote peer b0437c9edd764e29bc6fc894aba9f4c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:01.769037 4763 raft_consensus.cc:1261] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf [term 1 FOLLOWER]: Refusing update from remote peer b0437c9edd764e29bc6fc894aba9f4c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:01.769815 4966 consensus_queue.cc:1035] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90bce0a16c23458b9afdead97aa131cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:01.770092 4969 consensus_queue.cc:1035] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21431b6d1a86497a8d90dad419436f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:01.778378 4973 mvcc.cc:205] Tried to move back new op lower bound from 7298399853638623232 to 7298399853405065216. Current Snapshot: MvccSnapshot[applied={T|T < 7298399853638623232}]
I20260619 02:25:01.778810 4974 mvcc.cc:205] Tried to move back new op lower bound from 7298399853638623232 to 7298399853405065216. Current Snapshot: MvccSnapshot[applied={T|T < 7298399853638623232}]
I20260619 02:25:01.793640 4484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:01.793815 4484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:01.796108 4484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bdc73a1aa96d434a95255c292df1a763: Sending DeleteTablet for 3 replicas of tablet 1c8f068eb3204214815ba74b4c586e14
I20260619 02:25:01.796888 4876 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8f068eb3204214815ba74b4c586e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37606
I20260619 02:25:01.796965 4743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8f068eb3204214815ba74b4c586e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59462
I20260619 02:25:01.797369 4982 tablet_replica.cc:330] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9: stopping tablet replica
I20260619 02:25:01.797371 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8f068eb3204214815ba74b4c586e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47342
I20260619 02:25:01.797808 4983 tablet_replica.cc:330] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf: stopping tablet replica
I20260619 02:25:01.797871 4982 raft_consensus.cc:2229] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:25:01.798130 4983 raft_consensus.cc:2229] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:01.798390 4982 raft_consensus.cc:2258] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:01.798449 4983 raft_consensus.cc:2258] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:01.798651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 4548
I20260619 02:25:01.800006 4982 ts_tablet_manager.cc:1905] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:01.800006 4983 ts_tablet_manager.cc:1905] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:01.802646 4982 ts_tablet_manager.cc:1918] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:01.802646 4983 ts_tablet_manager.cc:1918] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:01.802738 4983 log.cc:1196] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/1c8f068eb3204214815ba74b4c586e14
I20260619 02:25:01.802760 4982 log.cc:1196] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/1c8f068eb3204214815ba74b4c586e14
I20260619 02:25:01.803087 4982 ts_tablet_manager.cc:1939] T 1c8f068eb3204214815ba74b4c586e14 P b0437c9edd764e29bc6fc894aba9f4c9: Deleting consensus metadata
I20260619 02:25:01.803087 4983 ts_tablet_manager.cc:1939] T 1c8f068eb3204214815ba74b4c586e14 P 90bce0a16c23458b9afdead97aa131cf: Deleting consensus metadata
I20260619 02:25:01.804208 4470 catalog_manager.cc:4841] TS 90bce0a16c23458b9afdead97aa131cf (127.0.98.2:36843): tablet 1c8f068eb3204214815ba74b4c586e14 (table test-table [id=3764cd320c3c49679412b6dd70e5a42a]) successfully deleted
I20260619 02:25:01.804323 4471 catalog_manager.cc:4841] TS b0437c9edd764e29bc6fc894aba9f4c9 (127.0.98.3:40399): tablet 1c8f068eb3204214815ba74b4c586e14 (table test-table [id=3764cd320c3c49679412b6dd70e5a42a]) successfully deleted
W20260619 02:25:01.809489 4470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34155 (error 108)
I20260619 02:25:01.810335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 4681
I20260619 02:25:01.810540 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:01.810245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34155 (local address 127.0.0.1:47348)
0619 02:25:01.810366 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:01.810438 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 02:25:01.811347 4470 catalog_manager.cc:4584] TS 21431b6d1a86497a8d90dad419436f0c (127.0.98.1:34155): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c8f068eb3204214815ba74b4c586e14: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34155: connect: Connection refused (error 111)
I20260619 02:25:01.817116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 4814
I20260619 02:25:01.823016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 4455
2026-06-19T02:25:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2382 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T02:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:01Z Disabled control of system clock
I20260619 02:25:01.853724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40495
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 02:25:01.961215 4989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:01.961479 4989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:01.961516 4989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:01.965516 4989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:01.965593 4989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:01.965610 4989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:01.965626 4989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:01.965641 4989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:25:01.970592 4989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40495
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:01.971580 4989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:01.972858 4989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:01.981194 4997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:01.981254 4989 server_base.cc:1034] running on GCE node
W20260619 02:25:01.981199 4994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:01.981223 4995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:01.982069 4989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:01.983273 4989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:01.984488 4989 hybrid_clock.cc:648] HybridClock initialized: now 1781835901984477 us; error 29 us; skew 500 ppm
I20260619 02:25:01.986675 4989 webserver.cc:460] Webserver started at http://127.0.98.62:33715/ using document root <none> and password file <none>
I20260619 02:25:01.987317 4989 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:01.987427 4989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:01.987664 4989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:01.989959 4989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "b802bd30962a4e448d25898e95b35e77"
format_stamp: "Formatted at 2026-06-19 02:25:01 on dist-test-slave-xkrt"
server_key: "d2ae93b1ffdc2fcb51540584d86bc22b"
server_key_iv: "f4a1138b17b684488fe7920aedffff74"
server_key_version: "encryptionkey@0"
I20260619 02:25:01.990598 4989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "b802bd30962a4e448d25898e95b35e77"
format_stamp: "Formatted at 2026-06-19 02:25:01 on dist-test-slave-xkrt"
server_key: "d2ae93b1ffdc2fcb51540584d86bc22b"
server_key_iv: "f4a1138b17b684488fe7920aedffff74"
server_key_version: "encryptionkey@0"
I20260619 02:25:01.994503 4989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:25:01.996927 5003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:01.998210 4989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:25:01.998380 4989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "b802bd30962a4e448d25898e95b35e77"
format_stamp: "Formatted at 2026-06-19 02:25:01 on dist-test-slave-xkrt"
server_key: "d2ae93b1ffdc2fcb51540584d86bc22b"
server_key_iv: "f4a1138b17b684488fe7920aedffff74"
server_key_version: "encryptionkey@0"
I20260619 02:25:01.998497 4989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:02.002869 4989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:02.003466 4989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:02.003641 4989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:02.010496 4989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40495
I20260619 02:25:02.010514 5055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40495 every 8 connection(s)
I20260619 02:25:02.011384 4989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:25:02.014504 5056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:02.018896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 4989
I20260619 02:25:02.019032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:25:02.019349 392 external_mini_cluster.cc:1428] Setting key f884b99bd5f605e17b7e2faef241e801
I20260619 02:25:02.020221 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77: Bootstrap starting.
I20260619 02:25:02.022722 5056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:02.023559 5056 log.cc:824] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:02.025308 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.020842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.0.98.62:40495)
0619 02:25:02.021484 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:02.021498 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:02.021533 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:25:02.022225 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:02.022242 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:02.022637 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:02.023048 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:02.023841 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:02.023846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:02.023857 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:02.023867 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:02.023949 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:02.023974 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:02.024088 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:02.024231 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 02:25:02.024475 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":123,"threads_started":1}
I20260619 02:25:02.025731 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77: No bootstrap required, opened a new log
I20260619 02:25:02.029412 5056 raft_consensus.cc:348] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b802bd30962a4e448d25898e95b35e77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } }
I20260619 02:25:02.029687 5056 raft_consensus.cc:374] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:02.029729 5056 raft_consensus.cc:729] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b802bd30962a4e448d25898e95b35e77, State: Initialized, Role: FOLLOWER
I20260619 02:25:02.030081 5056 consensus_queue.cc:260] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b802bd30962a4e448d25898e95b35e77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } }
I20260619 02:25:02.030187 5056 raft_consensus.cc:388] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:25:02.030225 5056 raft_consensus.cc:482] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:25:02.030292 5056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:02.031116 5056 raft_consensus.cc:504] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b802bd30962a4e448d25898e95b35e77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } }
I20260619 02:25:02.031407 5056 leader_election.cc:302] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b802bd30962a4e448d25898e95b35e77; no voters:
I20260619 02:25:02.031667 5056 leader_election.cc:288] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:25:02.031782 5061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:02.032006 5061 raft_consensus.cc:686] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [term 1 LEADER]: Becoming Leader. State: Replica: b802bd30962a4e448d25898e95b35e77, State: Running, Role: LEADER
I20260619 02:25:02.032339 5061 consensus_queue.cc:237] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b802bd30962a4e448d25898e95b35e77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } }
I20260619 02:25:02.033001 5056 sys_catalog.cc:564] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:02.034195 5063 sys_catalog.cc:455] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b802bd30962a4e448d25898e95b35e77. Latest consensus state: current_term: 1 leader_uuid: "b802bd30962a4e448d25898e95b35e77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b802bd30962a4e448d25898e95b35e77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } }
I20260619 02:25:02.034197 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b802bd30962a4e448d25898e95b35e77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b802bd30962a4e448d25898e95b35e77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } }
I20260619 02:25:02.034449 5063 sys_catalog.cc:458] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:02.034484 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:02.034870 5070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:02.037578 5070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:02.043190 5070 catalog_manager.cc:1347] Generated new cluster ID: 9e3645989ffa43c694bd98a0699e7b38
I20260619 02:25:02.043373 5070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:02.064337 5070 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:25:02.064664 5070 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:02.073609 5070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77: Generated new TSK 0
I20260619 02:25:02.074416 5070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:25:02.082515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40495
--builtin_ntp_servers=127.0.98.20:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 02:25:02.193945 5080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:02.194192 5080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:02.194270 5080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:02.198299 5080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:02.198361 5080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:02.198463 5080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:25:02.203365 5080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:02.204439 5080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:02.205825 5080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:02.215407 5088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:02.215488 5080 server_base.cc:1034] running on GCE node
W20260619 02:25:02.215396 5086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:02.215572 5085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:02.216080 5080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:02.216688 5080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:02.217845 5080 hybrid_clock.cc:648] HybridClock initialized: now 1781835902217828 us; error 29 us; skew 500 ppm
I20260619 02:25:02.219635 5080 webserver.cc:460] Webserver started at http://127.0.98.1:45303/ using document root <none> and password file <none>
I20260619 02:25:02.220170 5080 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:02.220232 5080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:02.220408 5080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:02.222472 5080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "9eb7312806144e17844c054bb514f9e6"
format_stamp: "Formatted at 2026-06-19 02:25:02 on dist-test-slave-xkrt"
server_key: "e4bbb61b2c426fcfcd24a4af134a0d16"
server_key_iv: "2bc8b4d8c270c8bed930ed60b824300d"
server_key_version: "encryptionkey@0"
I20260619 02:25:02.222970 5080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9eb7312806144e17844c054bb514f9e6"
format_stamp: "Formatted at 2026-06-19 02:25:02 on dist-test-slave-xkrt"
server_key: "e4bbb61b2c426fcfcd24a4af134a0d16"
server_key_iv: "2bc8b4d8c270c8bed930ed60b824300d"
server_key_version: "encryptionkey@0"
I20260619 02:25:02.226783 5080 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:25:02.229149 5094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:02.230764 5080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:25:02.230859 5080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "9eb7312806144e17844c054bb514f9e6"
format_stamp: "Formatted at 2026-06-19 02:25:02 on dist-test-slave-xkrt"
server_key: "e4bbb61b2c426fcfcd24a4af134a0d16"
server_key_iv: "2bc8b4d8c270c8bed930ed60b824300d"
server_key_version: "encryptionkey@0"
I20260619 02:25:02.230947 5080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:02.235491 5080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:02.236018 5080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:02.236150 5080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:02.236617 5080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:02.237437 5080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:02.237483 5080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:02.237519 5080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:02.237533 5080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:02.246271 5080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44167
I20260619 02:25:02.246302 5207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44167 every 8 connection(s)
I20260619 02:25:02.247108 5080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:25:02.248795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 5080
I20260619 02:25:02.248924 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:25:02.249205 392 external_mini_cluster.cc:1428] Setting key ce919c31066845e5e70e8e853960273c
I20260619 02:25:02.251330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40495
--builtin_ntp_servers=127.0.98.20:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 02:25:02.253916 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.248874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36151 (local address 127.0.98.62:40495)
0619 02:25:02.249032 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:02.249036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:02.250273 (+ 1237us) server_negotiation.cc:407] Connection header received
0619 02:25:02.251998 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:02.252001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:02.252035 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:02.252105 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:02.253343 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:02.253349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:02.253354 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:02.253370 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:02.253395 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:02.253408 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:02.253504 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:02.253699 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 02:25:02.253744 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:25:02.254626 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.249342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40495 (local address 127.0.98.1:36151)
0619 02:25:02.250134 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:02.250174 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:02.251764 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:02.252276 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:02.252287 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:02.252398 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:02.253087 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:25:02.253089 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:02.253126 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:25:02.253191 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:02.253498 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:02.253504 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:02.253508 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:25:02.253690 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 02:25:02.253868 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":183,"threads_started":1}
I20260619 02:25:02.255894 5208 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40495
I20260619 02:25:02.256126 5208 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:02.256618 5208 heartbeater.cc:511] Master 127.0.98.62:40495 requested a full tablet report, sending...
I20260619 02:25:02.258173 5020 ts_manager.cc:194] Registered new tserver with Master: 9eb7312806144e17844c054bb514f9e6 (127.0.98.1:44167)
W20260619 02:25:02.363204 5211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:02.363675 5211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:02.363794 5211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:02.367884 5211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:02.367946 5211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:02.368002 5211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:25:02.372694 5211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:02.373746 5211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:02.375067 5211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:02.384629 5217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:02.384635 5219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:02.384635 5216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:02.385032 5211 server_base.cc:1034] running on GCE node
I20260619 02:25:02.385449 5211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:02.386086 5211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:02.387246 5211 hybrid_clock.cc:648] HybridClock initialized: now 1781835902387212 us; error 46 us; skew 500 ppm
I20260619 02:25:02.389052 5211 webserver.cc:460] Webserver started at http://127.0.98.2:45797/ using document root <none> and password file <none>
I20260619 02:25:02.389667 5211 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:02.389731 5211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:02.389937 5211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:02.391893 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "81e94cff11274b48ba85fd0304bfe807"
format_stamp: "Formatted at 2026-06-19 02:25:02 on dist-test-slave-xkrt"
server_key: "4199a25c31936a88a34e6fcac1b297f3"
server_key_iv: "5c657a76285fce0f254320b1496ffb07"
server_key_version: "encryptionkey@0"
I20260619 02:25:02.392464 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "81e94cff11274b48ba85fd0304bfe807"
format_stamp: "Formatted at 2026-06-19 02:25:02 on dist-test-slave-xkrt"
server_key: "4199a25c31936a88a34e6fcac1b297f3"
server_key_iv: "5c657a76285fce0f254320b1496ffb07"
server_key_version: "encryptionkey@0"
I20260619 02:25:02.396046 5211 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:25:02.398499 5225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:02.399698 5211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:02.399852 5211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "81e94cff11274b48ba85fd0304bfe807"
format_stamp: "Formatted at 2026-06-19 02:25:02 on dist-test-slave-xkrt"
server_key: "4199a25c31936a88a34e6fcac1b297f3"
server_key_iv: "5c657a76285fce0f254320b1496ffb07"
server_key_version: "encryptionkey@0"
I20260619 02:25:02.399979 5211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:02.404629 5211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:02.405226 5211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:02.405402 5211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:02.405993 5211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:02.406852 5211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:02.406921 5211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:02.406977 5211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:02.407029 5211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:02.416056 5211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36733
I20260619 02:25:02.416086 5338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36733 every 8 connection(s)
I20260619 02:25:02.416913 5211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:25:02.418898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 5211
I20260619 02:25:02.419040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:25:02.419337 392 external_mini_cluster.cc:1428] Setting key 6bb388761bb940a2896445e0eb98bdd9
I20260619 02:25:02.421334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40495
--builtin_ntp_servers=127.0.98.20:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 02:25:02.423522 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.418784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47101 (local address 127.0.98.62:40495)
0619 02:25:02.418920 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:02.418923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:02.420233 (+ 1310us) server_negotiation.cc:407] Connection header received
0619 02:25:02.421784 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:02.421787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:02.421828 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:02.421909 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:02.423030 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:02.423034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:02.423036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:02.423048 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:02.423070 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:02.423080 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:02.423179 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:02.423348 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 02:25:02.423392 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:25:02.426322 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.419290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40495 (local address 127.0.98.2:47101)
0619 02:25:02.420063 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:02.420125 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:02.421531 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:02.422066 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:02.422073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:02.422159 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:02.422770 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:25:02.422772 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:02.422810 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:25:02.422876 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:02.423177 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:02.423187 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:02.423196 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:25:02.423388 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 02:25:02.425621 (+ 2233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":148,"threads_started":1}
I20260619 02:25:02.427496 5339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40495
I20260619 02:25:02.427764 5339 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:02.428252 5339 heartbeater.cc:511] Master 127.0.98.62:40495 requested a full tablet report, sending...
I20260619 02:25:02.429325 5020 ts_manager.cc:194] Registered new tserver with Master: 81e94cff11274b48ba85fd0304bfe807 (127.0.98.2:36733)
W20260619 02:25:02.530381 5342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:02.530664 5342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:02.530736 5342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:02.534909 5342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:02.534991 5342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:02.535116 5342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:25:02.539968 5342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:02.541217 5342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:02.543159 5342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:02.555254 5347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:02.555285 5350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:02.555269 5348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:02.555765 5342 server_base.cc:1034] running on GCE node
I20260619 02:25:02.556244 5342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:02.557108 5342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:02.558279 5342 hybrid_clock.cc:648] HybridClock initialized: now 1781835902558201 us; error 94 us; skew 500 ppm
I20260619 02:25:02.560098 5342 webserver.cc:460] Webserver started at http://127.0.98.3:40667/ using document root <none> and password file <none>
I20260619 02:25:02.560660 5342 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:02.560721 5342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:02.560930 5342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:02.562961 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "fb7058fd78a044bf845f335ab87c4e86"
format_stamp: "Formatted at 2026-06-19 02:25:02 on dist-test-slave-xkrt"
server_key: "558f7382c914b18f8886ad72625e7dd6"
server_key_iv: "a295e310d15e76bca732cc2ada5967b0"
server_key_version: "encryptionkey@0"
I20260619 02:25:02.563522 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fb7058fd78a044bf845f335ab87c4e86"
format_stamp: "Formatted at 2026-06-19 02:25:02 on dist-test-slave-xkrt"
server_key: "558f7382c914b18f8886ad72625e7dd6"
server_key_iv: "a295e310d15e76bca732cc2ada5967b0"
server_key_version: "encryptionkey@0"
I20260619 02:25:02.567162 5342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:25:02.569492 5356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:02.570699 5342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:02.570832 5342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "fb7058fd78a044bf845f335ab87c4e86"
format_stamp: "Formatted at 2026-06-19 02:25:02 on dist-test-slave-xkrt"
server_key: "558f7382c914b18f8886ad72625e7dd6"
server_key_iv: "a295e310d15e76bca732cc2ada5967b0"
server_key_version: "encryptionkey@0"
I20260619 02:25:02.570957 5342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:02.575389 5342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:02.575976 5342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:02.576157 5342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:02.576687 5342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:02.577623 5342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:02.577688 5342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:02.577745 5342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:02.577797 5342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:02.586309 5342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45879
I20260619 02:25:02.586366 5469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45879 every 8 connection(s)
I20260619 02:25:02.587219 5342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:25:02.590278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 5342
I20260619 02:25:02.590409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:25:02.590704 392 external_mini_cluster.cc:1428] Setting key 7fa559a8e33e9ba5a2ac8758487457fc
I20260619 02:25:02.594202 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.589124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54015 (local address 127.0.98.62:40495)
0619 02:25:02.589308 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:02.589313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:02.590734 (+ 1421us) server_negotiation.cc:407] Connection header received
0619 02:25:02.592258 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:02.592262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:02.592310 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:02.592384 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:02.593663 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:02.593667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:02.593670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:02.593691 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:02.593711 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:02.593720 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:02.593831 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:02.594014 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 02:25:02.594062 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:25:02.595080 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.589729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40495 (local address 127.0.98.3:54015)
0619 02:25:02.590555 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:02.590611 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:02.592069 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:02.592556 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:02.592565 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:02.592680 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:02.593347 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:25:02.593350 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:02.593395 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:25:02.593468 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:02.593835 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:02.593847 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:02.593855 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:25:02.594046 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 02:25:02.594303 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":118,"threads_started":1}
I20260619 02:25:02.596277 5470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40495
I20260619 02:25:02.596485 5470 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:02.596853 5470 heartbeater.cc:511] Master 127.0.98.62:40495 requested a full tablet report, sending...
I20260619 02:25:02.597827 5020 ts_manager.cc:194] Registered new tserver with Master: fb7058fd78a044bf845f335ab87c4e86 (127.0.98.3:45879)
I20260619 02:25:02.604996 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:25:02.608618 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.606920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.0.98.62:40495)
0619 02:25:02.607071 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:02.607075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:02.607271 (+ 196us) server_negotiation.cc:407] Connection header received
0619 02:25:02.607427 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:02.607434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:02.607469 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:02.607551 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:02.608229 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:02.608232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:02.608235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:02.608246 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:02.608266 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:02.608282 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:02.608370 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:02.608448 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 02:25:02.608489 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:25:02.611405 5020 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:25:02.614564 5020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:25:02.625981 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.623368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36733 (local address 127.0.0.1:34432)
0619 02:25:02.623808 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:02.623822 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:02.623899 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:02.624550 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:02.624553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:02.624568 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:02.625106 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:25:02.625108 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:02.625119 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:25:02.625155 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:02.625779 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:02.625782 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:02.625785 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:25:02.625868 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:25:02.625894 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":95,"threads_started":1}
I20260619 02:25:02.626132 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.623753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.0.98.2:36733)
0619 02:25:02.624128 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:02.624141 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:02.624165 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:25:02.624217 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:02.624223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:02.624393 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:02.624766 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:02.625594 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:02.625601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:02.625613 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:02.625625 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:02.625661 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:02.625684 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:02.625793 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:02.625884 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 02:25:02.625943 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":148,"threads_started":1}
I20260619 02:25:02.626349 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.623347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54960 (local address 127.0.98.3:45879)
0619 02:25:02.623809 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:02.623823 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:02.623851 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:25:02.623917 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:02.623923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:02.624129 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:02.624272 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:02.625400 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:02.625406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:02.625418 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:02.625430 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:02.625471 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:02.625498 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:02.626060 (+ 562us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:02.626109 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 02:25:02.626177 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":148,"threads_started":1}
I20260619 02:25:02.625982 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.622751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45879 (local address 127.0.0.1:54960)
0619 02:25:02.623461 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:02.623482 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:02.623580 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:02.624314 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:02.624319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:02.624402 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:02.624968 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:25:02.624972 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:02.624996 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:25:02.625068 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:02.625627 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:02.625636 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:02.625642 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:25:02.625754 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:25:02.625825 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":81,"threads_started":1}
I20260619 02:25:02.626968 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.622781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.0.98.1:44167)
0619 02:25:02.623079 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:02.623091 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:02.624180 (+ 1089us) server_negotiation.cc:407] Connection header received
0619 02:25:02.624267 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:02.624271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:02.624402 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:02.624525 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:02.626336 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:02.626342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:02.626370 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:02.626385 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:02.626428 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:02.626469 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:02.626572 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:02.626707 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 02:25:02.626775 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":117,"threads_started":1}
I20260619 02:25:02.627255 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.622641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44167 (local address 127.0.0.1:41642)
0619 02:25:02.623293 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:02.624099 (+ 806us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:02.624182 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:02.625174 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:02.625180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:02.625198 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:02.625647 (+ 449us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:25:02.625649 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:02.625660 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:25:02.625684 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:02.626580 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:02.626585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:02.626591 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:25:02.627135 (+ 544us) client_negotiation.cc:240] Negotiation successful
0619 02:25:02.627176 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":80,"threads_started":1}
I20260619 02:25:02.628926 5273 tablet_service.cc:1467] Processing CreateTablet for tablet b58681da56bf4bcea2e990a9a7f8ed35 (DEFAULT_TABLE table=test-table [id=01525dff9a1645dd814ed83c0bbc30e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:02.629931 5273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58681da56bf4bcea2e990a9a7f8ed35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:02.629738 5404 tablet_service.cc:1467] Processing CreateTablet for tablet b58681da56bf4bcea2e990a9a7f8ed35 (DEFAULT_TABLE table=test-table [id=01525dff9a1645dd814ed83c0bbc30e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:02.630717 5404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58681da56bf4bcea2e990a9a7f8ed35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:02.630852 5142 tablet_service.cc:1467] Processing CreateTablet for tablet b58681da56bf4bcea2e990a9a7f8ed35 (DEFAULT_TABLE table=test-table [id=01525dff9a1645dd814ed83c0bbc30e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:02.631791 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58681da56bf4bcea2e990a9a7f8ed35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:02.636291 5487 tablet_bootstrap.cc:492] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807: Bootstrap starting.
I20260619 02:25:02.637668 5488 tablet_bootstrap.cc:492] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86: Bootstrap starting.
I20260619 02:25:02.638832 5489 tablet_bootstrap.cc:492] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6: Bootstrap starting.
I20260619 02:25:02.639343 5487 tablet_bootstrap.cc:654] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:02.639724 5488 tablet_bootstrap.cc:654] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:02.640226 5487 log.cc:824] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:02.640465 5488 log.cc:824] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:02.640722 5489 tablet_bootstrap.cc:654] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:02.641508 5489 log.cc:824] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:02.642431 5488 tablet_bootstrap.cc:492] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86: No bootstrap required, opened a new log
I20260619 02:25:02.642464 5487 tablet_bootstrap.cc:492] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807: No bootstrap required, opened a new log
I20260619 02:25:02.642614 5488 ts_tablet_manager.cc:1397] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:25:02.642634 5487 ts_tablet_manager.cc:1397] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:25:02.643340 5489 tablet_bootstrap.cc:492] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6: No bootstrap required, opened a new log
I20260619 02:25:02.643508 5489 ts_tablet_manager.cc:1397] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 02:25:02.646293 5487 raft_consensus.cc:348] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb7312806144e17844c054bb514f9e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44167 } } peers { permanent_uuid: "fb7058fd78a044bf845f335ab87c4e86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45879 } } peers { permanent_uuid: "81e94cff11274b48ba85fd0304bfe807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36733 } }
I20260619 02:25:02.646543 5487 raft_consensus.cc:374] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:02.646605 5487 raft_consensus.cc:729] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e94cff11274b48ba85fd0304bfe807, State: Initialized, Role: FOLLOWER
I20260619 02:25:02.647044 5487 consensus_queue.cc:260] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb7312806144e17844c054bb514f9e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44167 } } peers { permanent_uuid: "fb7058fd78a044bf845f335ab87c4e86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45879 } } peers { permanent_uuid: "81e94cff11274b48ba85fd0304bfe807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36733 } }
I20260619 02:25:02.647190 5488 raft_consensus.cc:348] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb7312806144e17844c054bb514f9e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44167 } } peers { permanent_uuid: "fb7058fd78a044bf845f335ab87c4e86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45879 } } peers { permanent_uuid: "81e94cff11274b48ba85fd0304bfe807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36733 } }
I20260619 02:25:02.647449 5488 raft_consensus.cc:374] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:02.647507 5488 raft_consensus.cc:729] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb7058fd78a044bf845f335ab87c4e86, State: Initialized, Role: FOLLOWER
I20260619 02:25:02.647796 5339 heartbeater.cc:503] Master 127.0.98.62:40495 was elected leader, sending a full tablet report...
I20260619 02:25:02.647812 5489 raft_consensus.cc:348] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb7312806144e17844c054bb514f9e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44167 } } peers { permanent_uuid: "fb7058fd78a044bf845f335ab87c4e86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45879 } } peers { permanent_uuid: "81e94cff11274b48ba85fd0304bfe807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36733 } }
I20260619 02:25:02.647922 5488 consensus_queue.cc:260] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb7312806144e17844c054bb514f9e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44167 } } peers { permanent_uuid: "fb7058fd78a044bf845f335ab87c4e86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45879 } } peers { permanent_uuid: "81e94cff11274b48ba85fd0304bfe807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36733 } }
I20260619 02:25:02.648037 5489 raft_consensus.cc:374] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:02.648053 5487 ts_tablet_manager.cc:1428] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:25:02.648079 5489 raft_consensus.cc:729] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb7312806144e17844c054bb514f9e6, State: Initialized, Role: FOLLOWER
I20260619 02:25:02.648460 5489 consensus_queue.cc:260] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb7312806144e17844c054bb514f9e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44167 } } peers { permanent_uuid: "fb7058fd78a044bf845f335ab87c4e86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45879 } } peers { permanent_uuid: "81e94cff11274b48ba85fd0304bfe807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36733 } }
I20260619 02:25:02.648908 5488 ts_tablet_manager.cc:1428] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260619 02:25:02.649245 5470 heartbeater.cc:503] Master 127.0.98.62:40495 was elected leader, sending a full tablet report...
I20260619 02:25:02.649297 5489 ts_tablet_manager.cc:1428] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:25:02.650158 5208 heartbeater.cc:503] Master 127.0.98.62:40495 was elected leader, sending a full tablet report...
W20260619 02:25:02.667819 5340 tablet.cc:2369] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:25:02.748172 5209 tablet.cc:2369] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:25:02.760553 5493 raft_consensus.cc:482] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:25:02.760776 5493 raft_consensus.cc:504] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb7312806144e17844c054bb514f9e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44167 } } peers { permanent_uuid: "fb7058fd78a044bf845f335ab87c4e86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45879 } } peers { permanent_uuid: "81e94cff11274b48ba85fd0304bfe807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36733 } }
I20260619 02:25:02.762014 5493 leader_election.cc:288] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9eb7312806144e17844c054bb514f9e6 (127.0.98.1:44167), fb7058fd78a044bf845f335ab87c4e86 (127.0.98.3:45879)
I20260619 02:25:02.764134 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.762283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45879 (local address 127.0.98.2:60291)
0619 02:25:02.762402 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:02.762418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:02.762507 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:02.762769 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:02.762773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:02.762791 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:02.763511 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:25:02.763513 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:02.763526 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:25:02.763553 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:02.763879 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:02.763883 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:02.763886 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:25:02.763990 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 02:25:02.764029 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 02:25:02.764222 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.762386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60291 (local address 127.0.98.3:45879)
0619 02:25:02.762507 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:02.762512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:02.762530 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:25:02.762585 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:02.762587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:02.762625 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:02.762710 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:02.763701 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:02.763707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:02.763714 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:02.763730 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:02.763756 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:02.763770 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:02.763874 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:02.764014 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 02:25:02.764060 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:25:02.764608 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.762386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58699 (local address 127.0.98.1:44167)
0619 02:25:02.762508 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:02.762511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:02.763044 (+ 533us) server_negotiation.cc:407] Connection header received
0619 02:25:02.763110 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:02.763115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:02.763157 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:02.763278 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:02.764054 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:02.764058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:02.764060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:02.764072 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:02.764092 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:02.764102 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:02.764194 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:02.764450 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 02:25:02.764491 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:25:02.764627 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.762300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44167 (local address 127.0.98.2:58699)
0619 02:25:02.762900 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:02.762916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:02.763011 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:02.763282 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:02.763284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:02.763296 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:02.763896 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:25:02.763897 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:02.763909 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:25:02.763939 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:02.764198 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:02.764334 (+ 136us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:02.764339 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:25:02.764456 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 02:25:02.764489 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":273,"thread_start_us":152,"threads_started":1}
I20260619 02:25:02.765033 5424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58681da56bf4bcea2e990a9a7f8ed35" candidate_uuid: "81e94cff11274b48ba85fd0304bfe807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb7058fd78a044bf845f335ab87c4e86" is_pre_election: true
I20260619 02:25:02.765316 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58681da56bf4bcea2e990a9a7f8ed35" candidate_uuid: "81e94cff11274b48ba85fd0304bfe807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb7312806144e17844c054bb514f9e6" is_pre_election: true
I20260619 02:25:02.765378 5424 raft_consensus.cc:2454] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81e94cff11274b48ba85fd0304bfe807 in term 0.
I20260619 02:25:02.765628 5162 raft_consensus.cc:2454] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81e94cff11274b48ba85fd0304bfe807 in term 0.
I20260619 02:25:02.765951 5228 leader_election.cc:302] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81e94cff11274b48ba85fd0304bfe807, fb7058fd78a044bf845f335ab87c4e86; no voters:
I20260619 02:25:02.766227 5493 raft_consensus.cc:2781] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:25:02.766306 5493 raft_consensus.cc:482] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:25:02.766347 5493 raft_consensus.cc:3037] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:02.767371 5493 raft_consensus.cc:504] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb7312806144e17844c054bb514f9e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44167 } } peers { permanent_uuid: "fb7058fd78a044bf845f335ab87c4e86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45879 } } peers { permanent_uuid: "81e94cff11274b48ba85fd0304bfe807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36733 } }
I20260619 02:25:02.767704 5493 leader_election.cc:288] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [CANDIDATE]: Term 1 election: Requested vote from peers 9eb7312806144e17844c054bb514f9e6 (127.0.98.1:44167), fb7058fd78a044bf845f335ab87c4e86 (127.0.98.3:45879)
I20260619 02:25:02.768051 5424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58681da56bf4bcea2e990a9a7f8ed35" candidate_uuid: "81e94cff11274b48ba85fd0304bfe807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb7058fd78a044bf845f335ab87c4e86"
I20260619 02:25:02.768045 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58681da56bf4bcea2e990a9a7f8ed35" candidate_uuid: "81e94cff11274b48ba85fd0304bfe807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb7312806144e17844c054bb514f9e6"
I20260619 02:25:02.768195 5424 raft_consensus.cc:3037] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:02.768208 5162 raft_consensus.cc:3037] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:02.769222 5424 raft_consensus.cc:2454] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81e94cff11274b48ba85fd0304bfe807 in term 1.
I20260619 02:25:02.769276 5162 raft_consensus.cc:2454] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81e94cff11274b48ba85fd0304bfe807 in term 1.
I20260619 02:25:02.769519 5228 leader_election.cc:302] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81e94cff11274b48ba85fd0304bfe807, fb7058fd78a044bf845f335ab87c4e86; no voters:
I20260619 02:25:02.769698 5493 raft_consensus.cc:2781] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:02.769901 5493 raft_consensus.cc:686] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [term 1 LEADER]: Becoming Leader. State: Replica: 81e94cff11274b48ba85fd0304bfe807, State: Running, Role: LEADER
I20260619 02:25:02.770251 5493 consensus_queue.cc:237] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb7312806144e17844c054bb514f9e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44167 } } peers { permanent_uuid: "fb7058fd78a044bf845f335ab87c4e86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45879 } } peers { permanent_uuid: "81e94cff11274b48ba85fd0304bfe807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36733 } }
I20260619 02:25:02.773512 5019 catalog_manager.cc:5495] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81e94cff11274b48ba85fd0304bfe807 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "81e94cff11274b48ba85fd0304bfe807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb7312806144e17844c054bb514f9e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb7058fd78a044bf845f335ab87c4e86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81e94cff11274b48ba85fd0304bfe807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36733 } health_report { overall_health: HEALTHY } } }
I20260619 02:25:02.781697 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.780338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.0.98.2:36733)
0619 02:25:02.780460 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:02.780464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:02.780475 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:25:02.780598 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:02.780601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:02.780652 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:02.780719 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:02.781320 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:02.781325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:02.781332 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:02.781344 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:02.781362 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:02.781370 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:02.781451 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:02.781524 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 02:25:02.781595 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:25:02.787245 5162 raft_consensus.cc:1261] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6 [term 1 FOLLOWER]: Refusing update from remote peer 81e94cff11274b48ba85fd0304bfe807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:02.787245 5424 raft_consensus.cc:1261] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86 [term 1 FOLLOWER]: Refusing update from remote peer 81e94cff11274b48ba85fd0304bfe807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:02.787946 5497 consensus_queue.cc:1035] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9eb7312806144e17844c054bb514f9e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:02.788172 5493 consensus_queue.cc:1035] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb7058fd78a044bf845f335ab87c4e86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:02.795974 5501 mvcc.cc:205] Tried to move back new op lower bound from 7298399857809281024 to 7298399857749659648. Current Snapshot: MvccSnapshot[applied={T|T < 7298399857809281024}]
I20260619 02:25:02.804323 5019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:02.804507 5019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:02.806972 5019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b802bd30962a4e448d25898e95b35e77: Sending DeleteTablet for 3 replicas of tablet b58681da56bf4bcea2e990a9a7f8ed35
I20260619 02:25:02.807749 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet b58681da56bf4bcea2e990a9a7f8ed35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:02 UTC) from {username='slave'} at 127.0.0.1:41642
I20260619 02:25:02.807842 5273 tablet_service.cc:1514] Processing DeleteTablet for tablet b58681da56bf4bcea2e990a9a7f8ed35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:02 UTC) from {username='slave'} at 127.0.0.1:34432
I20260619 02:25:02.807842 5404 tablet_service.cc:1514] Processing DeleteTablet for tablet b58681da56bf4bcea2e990a9a7f8ed35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:02 UTC) from {username='slave'} at 127.0.0.1:54960
I20260619 02:25:02.808248 5509 tablet_replica.cc:330] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6: stopping tablet replica
I20260619 02:25:02.808528 5511 tablet_replica.cc:330] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86: stopping tablet replica
I20260619 02:25:02.808560 5509 raft_consensus.cc:2229] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:02.808811 5511 raft_consensus.cc:2229] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:02.808902 5509 raft_consensus.cc:2258] T b58681da56bf4bcea2e990a9a7f8ed35 P 9eb7312806144e17844c054bb514f9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:02.809105 5511 raft_consensus.cc:2258] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:02.809146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 5080
I20260619 02:25:02.809437 5510 tablet_replica.cc:330] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807: stopping tablet replica
I20260619 02:25:02.810052 5510 raft_consensus.cc:2229] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:25:02.810842 5510 raft_consensus.cc:2258] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:02.811494 5511 ts_tablet_manager.cc:1905] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:02.812305 5510 ts_tablet_manager.cc:1905] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:02.814337 5511 ts_tablet_manager.cc:1918] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:02.814427 5511 log.cc:1196] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/b58681da56bf4bcea2e990a9a7f8ed35
I20260619 02:25:02.814740 5511 ts_tablet_manager.cc:1939] T b58681da56bf4bcea2e990a9a7f8ed35 P fb7058fd78a044bf845f335ab87c4e86: Deleting consensus metadata
I20260619 02:25:02.815689 5510 ts_tablet_manager.cc:1918] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:02.815773 5510 log.cc:1196] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/b58681da56bf4bcea2e990a9a7f8ed35
I20260619 02:25:02.815989 5006 catalog_manager.cc:4841] TS fb7058fd78a044bf845f335ab87c4e86 (127.0.98.3:45879): tablet b58681da56bf4bcea2e990a9a7f8ed35 (table test-table [id=01525dff9a1645dd814ed83c0bbc30e4]) successfully deleted
I20260619 02:25:02.816111 5510 ts_tablet_manager.cc:1939] T b58681da56bf4bcea2e990a9a7f8ed35 P 81e94cff11274b48ba85fd0304bfe807: Deleting consensus metadata
I20260619 02:25:02.817238 5006 catalog_manager.cc:4841] TS 81e94cff11274b48ba85fd0304bfe807 (127.0.98.2:36733): tablet b58681da56bf4bcea2e990a9a7f8ed35 (table test-table [id=01525dff9a1645dd814ed83c0bbc30e4]) successfully deleted
W20260619 02:25:02.817317 5007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44167 (error 108)
I20260619 02:25:02.818105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 5211
I20260619 02:25:02.818370 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:02.818067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44167 (local address 127.0.0.1:41644)
0619 02:25:02.818194 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:02.818266 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 02:25:02.819079 5007 catalog_manager.cc:4584] TS 9eb7312806144e17844c054bb514f9e6 (127.0.98.1:44167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b58681da56bf4bcea2e990a9a7f8ed35: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44167: connect: Connection refused (error 111)
I20260619 02:25:02.824028 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 5342
I20260619 02:25:02.829704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 4989
2026-06-19T02:25:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1005 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 02:25:02.850862 392 mini_oidc.cc:152] Starting JWKS server
I20260619 02:25:02.852442 392 webserver.cc:460] Webserver started at https://127.0.0.1:36863/ using document root <none> and password file <none>
I20260619 02:25:02.852614 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:25:02.852900 392 webserver.cc:460] Webserver started at http://127.0.0.1:42871/ using document root <none> and password file <none>
2026-06-19T02:25:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:02Z Disabled control of system clock
I20260619 02:25:02.862262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42537
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42537
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36863/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/testchainca.cert with env {}
W20260619 02:25:02.970952 5518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:02.971217 5518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:02.971266 5518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:02.975606 5518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:02.975668 5518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:02.975683 5518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:02.975699 5518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:02.975715 5518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:25:02.975777 5518 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:25:02.975795 5518 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36863/jwks.json
I20260619 02:25:02.981328 5518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36863/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:02.982442 5518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:02.983482 5518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:02.992425 5524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:02.992385 5526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:02.992616 5518 server_base.cc:1034] running on GCE node
W20260619 02:25:02.992412 5523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:02.993649 5518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:02.994867 5518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:02.996071 5518 hybrid_clock.cc:648] HybridClock initialized: now 1781835902996046 us; error 62 us; skew 500 ppm
I20260619 02:25:02.998127 5518 webserver.cc:460] Webserver started at http://127.0.98.62:44805/ using document root <none> and password file <none>
I20260619 02:25:02.998718 5518 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:02.998781 5518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:02.999002 5518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:03.001092 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "903fcc37fc05496ebc3d17f81279eb66"
format_stamp: "Formatted at 2026-06-19 02:25:02 on dist-test-slave-xkrt"
server_key: "3a8e3566c9c8a6800c73bc53a32ae474"
server_key_iv: "fb42cb9bc562d39dd74435d74b104324"
server_key_version: "encryptionkey@0"
I20260619 02:25:03.001690 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "903fcc37fc05496ebc3d17f81279eb66"
format_stamp: "Formatted at 2026-06-19 02:25:02 on dist-test-slave-xkrt"
server_key: "3a8e3566c9c8a6800c73bc53a32ae474"
server_key_iv: "fb42cb9bc562d39dd74435d74b104324"
server_key_version: "encryptionkey@0"
I20260619 02:25:03.005280 5518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:25:03.007766 5532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:03.009060 5518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:03.009202 5518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "903fcc37fc05496ebc3d17f81279eb66"
format_stamp: "Formatted at 2026-06-19 02:25:02 on dist-test-slave-xkrt"
server_key: "3a8e3566c9c8a6800c73bc53a32ae474"
server_key_iv: "fb42cb9bc562d39dd74435d74b104324"
server_key_version: "encryptionkey@0"
I20260619 02:25:03.009312 5518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:03.033567 5518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:03.034278 5518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:03.034482 5518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:03.041441 5518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42537
I20260619 02:25:03.041499 5584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42537 every 8 connection(s)
I20260619 02:25:03.042441 5518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:25:03.045773 5585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:03.047883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 5518
I20260619 02:25:03.048022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:25:03.048352 392 external_mini_cluster.cc:1428] Setting key 10a41f4ce3e28caa265996798900ce5e
I20260619 02:25:03.052834 5585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66: Bootstrap starting.
I20260619 02:25:03.055295 5585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:03.056106 5585 log.cc:824] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:03.057847 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:03.049831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.0.98.62:42537)
0619 02:25:03.050494 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:03.050509 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:03.050544 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:25:03.051196 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:03.051213 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:03.051595 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:03.051922 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:03.052377 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.053431 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.054336 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.054798 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.054981 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:03.055865 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:03.055870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:03.055891 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:03.056323 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:03.056377 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:03.056397 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:03.056515 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:03.056656 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 02:25:03.056933 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":153,"threads_started":1}
I20260619 02:25:03.058275 5585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66: No bootstrap required, opened a new log
I20260619 02:25:03.060801 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:25:03.062529 5585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "903fcc37fc05496ebc3d17f81279eb66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42537 } }
I20260619 02:25:03.062757 5585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:03.062832 5585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903fcc37fc05496ebc3d17f81279eb66, State: Initialized, Role: FOLLOWER
I20260619 02:25:03.063252 5585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "903fcc37fc05496ebc3d17f81279eb66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42537 } }
I20260619 02:25:03.063397 5585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:25:03.063488 5585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:25:03.063586 5585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:03.064585 5585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "903fcc37fc05496ebc3d17f81279eb66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42537 } }
I20260619 02:25:03.064869 5585 leader_election.cc:302] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 903fcc37fc05496ebc3d17f81279eb66; no voters:
I20260619 02:25:03.065104 5585 leader_election.cc:288] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:25:03.065222 5590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:03.065454 5590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [term 1 LEADER]: Becoming Leader. State: Replica: 903fcc37fc05496ebc3d17f81279eb66, State: Running, Role: LEADER
I20260619 02:25:03.065776 5590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "903fcc37fc05496ebc3d17f81279eb66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42537 } }
I20260619 02:25:03.066243 5585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:03.067342 5592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 903fcc37fc05496ebc3d17f81279eb66. Latest consensus state: current_term: 1 leader_uuid: "903fcc37fc05496ebc3d17f81279eb66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "903fcc37fc05496ebc3d17f81279eb66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42537 } } }
I20260619 02:25:03.067467 5592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:03.067716 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "903fcc37fc05496ebc3d17f81279eb66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "903fcc37fc05496ebc3d17f81279eb66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42537 } } }
I20260619 02:25:03.067817 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:03.068241 5599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:03.070847 5599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:03.076471 5599 catalog_manager.cc:1347] Generated new cluster ID: 953c84f0dd834635bc27d2015e2add9c
I20260619 02:25:03.076553 5599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:03.092680 5599 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:25:03.093937 5599 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:03.104920 5599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 903fcc37fc05496ebc3d17f81279eb66: Generated new TSK 0
I20260619 02:25:03.105674 5599 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 02:25:03.143745 5615 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 02:25:03.143857 5615 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 02:25:03.144361 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:03.119502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.0.98.62:42537)
0619 02:25:03.119644 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:03.119651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:03.119871 (+ 220us) server_negotiation.cc:407] Connection header received
0619 02:25:03.120123 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:03.120126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:03.120187 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:03.120332 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:25:03.120601 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.121189 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.122240 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.122507 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.122631 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:03.123091 (+ 460us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:25:03.143923 (+ 20832us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 02:25:03.144206 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35224: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 02:25:03.144461 5588 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35224: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 02:25:03.145689 5614 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:03.119426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42537 (local address 127.0.0.1:35224)
0619 02:25:03.119779 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:03.119794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:03.119951 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:03.120334 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:03.120338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:03.120347 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:25:03.120485 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:03.120492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.121315 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.121319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:03.122107 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:03.122116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.122618 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.122637 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:03.122744 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:03.122788 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:25:03.144154 (+ 21366us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 02:25:03.144469 (+ 315us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42537: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":107,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:42537: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 02:25:03.146660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 5518
2026-06-19T02:25:03Z chronyd exiting
I20260619 02:25:03.183712 392 test_util.cc:175] -----------------------------------------------
I20260619 02:25:03.183801 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781835873883564-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (334 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 02:25:03.184859 392 mini_oidc.cc:152] Starting JWKS server
I20260619 02:25:03.185813 392 webserver.cc:460] Webserver started at https://127.0.0.1:40557/ using document root <none> and password file <none>
I20260619 02:25:03.185925 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:25:03.186199 392 webserver.cc:460] Webserver started at http://127.0.0.1:45983/ using document root <none> and password file <none>
2026-06-19T02:25:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:03Z Disabled control of system clock
I20260619 02:25:03.195247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39529
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39529
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40557/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/testchainca.cert with env {}
W20260619 02:25:03.304121 5621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:03.304426 5621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:03.304500 5621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:03.308511 5621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:03.308604 5621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:03.308633 5621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:03.308673 5621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:03.308710 5621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:25:03.308754 5621 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:25:03.308787 5621 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40557/jwks.json
I20260619 02:25:03.314157 5621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39529
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40557/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:03.315225 5621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:03.316241 5621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:03.324916 5627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:03.324927 5629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:03.325006 5621 server_base.cc:1034] running on GCE node
W20260619 02:25:03.324916 5626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:03.325743 5621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:03.326866 5621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:03.328052 5621 hybrid_clock.cc:648] HybridClock initialized: now 1781835903328029 us; error 35 us; skew 500 ppm
I20260619 02:25:03.330083 5621 webserver.cc:460] Webserver started at http://127.0.98.62:45091/ using document root <none> and password file <none>
I20260619 02:25:03.330653 5621 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:03.330742 5621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:03.330958 5621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:03.333039 5621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "f1fda64ea87c422c9414e48a41146b9d"
format_stamp: "Formatted at 2026-06-19 02:25:03 on dist-test-slave-xkrt"
server_key: "d0f00c933b9b1eea8cd8e966588504a6"
server_key_iv: "c0013927cfa88bee110e44702790563a"
server_key_version: "encryptionkey@0"
I20260619 02:25:03.333669 5621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1fda64ea87c422c9414e48a41146b9d"
format_stamp: "Formatted at 2026-06-19 02:25:03 on dist-test-slave-xkrt"
server_key: "d0f00c933b9b1eea8cd8e966588504a6"
server_key_iv: "c0013927cfa88bee110e44702790563a"
server_key_version: "encryptionkey@0"
I20260619 02:25:03.337239 5621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:25:03.339730 5635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:03.341022 5621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:03.341161 5621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "f1fda64ea87c422c9414e48a41146b9d"
format_stamp: "Formatted at 2026-06-19 02:25:03 on dist-test-slave-xkrt"
server_key: "d0f00c933b9b1eea8cd8e966588504a6"
server_key_iv: "c0013927cfa88bee110e44702790563a"
server_key_version: "encryptionkey@0"
I20260619 02:25:03.341279 5621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:03.362840 5621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:03.363525 5621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:03.363721 5621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:03.370769 5621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39529
I20260619 02:25:03.370815 5687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39529 every 8 connection(s)
I20260619 02:25:03.371734 5621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:25:03.375044 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:03.380360 5688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d: Bootstrap starting.
I20260619 02:25:03.380913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 5621
I20260619 02:25:03.381047 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:25:03.381326 392 external_mini_cluster.cc:1428] Setting key fada26b911b134c0a6f2c34c72af2e8c
I20260619 02:25:03.383265 5688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:03.383988 5688 log.cc:824] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:03.386065 5688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d: No bootstrap required, opened a new log
I20260619 02:25:03.389694 5688 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1fda64ea87c422c9414e48a41146b9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39529 } }
I20260619 02:25:03.389918 5688 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:03.389992 5688 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1fda64ea87c422c9414e48a41146b9d, State: Initialized, Role: FOLLOWER
I20260619 02:25:03.390466 5688 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1fda64ea87c422c9414e48a41146b9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39529 } }
I20260619 02:25:03.390638 5688 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:25:03.390722 5688 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:25:03.390843 5688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:03.391610 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:03.382727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47292 (local address 127.0.98.62:39529)
0619 02:25:03.383454 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:03.383471 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:03.383510 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:25:03.384238 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:03.384259 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:03.384719 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:03.385134 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:03.385642 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.386828 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.387809 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.388332 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.388510 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:03.389486 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:03.389492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:03.389506 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:03.390096 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:03.390175 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:03.390197 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:03.390307 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:03.390488 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 02:25:03.390795 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
I20260619 02:25:03.391763 5688 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1fda64ea87c422c9414e48a41146b9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39529 } }
I20260619 02:25:03.392072 5688 leader_election.cc:302] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1fda64ea87c422c9414e48a41146b9d; no voters:
I20260619 02:25:03.392336 5688 leader_election.cc:288] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:25:03.392441 5693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:03.392632 5693 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [term 1 LEADER]: Becoming Leader. State: Replica: f1fda64ea87c422c9414e48a41146b9d, State: Running, Role: LEADER
I20260619 02:25:03.392903 5693 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1fda64ea87c422c9414e48a41146b9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39529 } }
I20260619 02:25:03.393570 5688 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:03.394318 5695 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1fda64ea87c422c9414e48a41146b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1fda64ea87c422c9414e48a41146b9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39529 } } }
I20260619 02:25:03.394474 5695 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:03.394747 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:25:03.394732 5694 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1fda64ea87c422c9414e48a41146b9d. Latest consensus state: current_term: 1 leader_uuid: "f1fda64ea87c422c9414e48a41146b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1fda64ea87c422c9414e48a41146b9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39529 } } }
I20260619 02:25:03.394817 5694 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:03.395054 5702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:03.398205 5702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:03.403914 5702 catalog_manager.cc:1347] Generated new cluster ID: 76609bf13da04895ba4838a60e3619df
I20260619 02:25:03.403976 5702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:03.415060 5702 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:25:03.416311 5702 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:03.430289 5702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1fda64ea87c422c9414e48a41146b9d: Generated new TSK 0
I20260619 02:25:03.431126 5702 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 02:25:03.462939 5718 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 02:25:03.463049 5718 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 02:25:03.463649 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:03.446943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.0.98.62:39529)
0619 02:25:03.447077 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:03.447081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:03.447325 (+ 244us) server_negotiation.cc:407] Connection header received
0619 02:25:03.447493 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:03.447496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:03.447553 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:03.447670 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:25:03.447925 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.448747 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.449710 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.450060 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.450158 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:03.450457 (+ 299us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:25:03.463248 (+ 12791us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 02:25:03.463502 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47306: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 02:25:03.463716 5691 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47306: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 02:25:03.463810 5717 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:03.446897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39529 (local address 127.0.0.1:47306)
0619 02:25:03.447210 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:03.447223 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:03.447371 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:03.447708 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:03.447711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:03.447718 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:25:03.447817 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:03.447822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.448873 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.448876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:03.449515 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:03.449523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.450165 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.450182 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:03.450260 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:03.450290 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:25:03.463470 (+ 13180us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 02:25:03.463615 (+ 145us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39529: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":141,"threads_started":1}
I20260619 02:25:03.464536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 5621
2026-06-19T02:25:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (318 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 02:25:03.503654 392 mini_oidc.cc:152] Starting JWKS server
I20260619 02:25:03.504513 392 webserver.cc:460] Webserver started at https://127.0.0.1:37673/ using document root <none> and password file <none>
I20260619 02:25:03.504629 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:25:03.504882 392 webserver.cc:460] Webserver started at http://127.0.0.1:37659/ using document root <none> and password file <none>
2026-06-19T02:25:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:03Z Disabled control of system clock
I20260619 02:25:03.513376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42393
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42393
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37673/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 02:25:03.622278 5724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:03.622555 5724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:03.622620 5724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:03.626713 5724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:03.626796 5724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:03.626825 5724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:03.626883 5724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:03.626921 5724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:25:03.626958 5724 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:25:03.626989 5724 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37673/jwks.json
I20260619 02:25:03.631726 5724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37673/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:03.632814 5724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:03.634150 5724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:03.642360 5729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:03.642393 5730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:03.642441 5724 server_base.cc:1034] running on GCE node
W20260619 02:25:03.642393 5732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:03.643139 5724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:03.644225 5724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:03.645428 5724 hybrid_clock.cc:648] HybridClock initialized: now 1781835903645414 us; error 28 us; skew 500 ppm
I20260619 02:25:03.647313 5724 webserver.cc:460] Webserver started at http://127.0.98.62:38379/ using document root <none> and password file <none>
I20260619 02:25:03.647881 5724 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:03.647970 5724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:03.648195 5724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:03.650285 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "2556aa76b8a442ef84a5ed3fad5f7189"
format_stamp: "Formatted at 2026-06-19 02:25:03 on dist-test-slave-xkrt"
server_key: "85e5eb4b308a21300904b80a0cfaaec9"
server_key_iv: "7c12580c2da3558e71a919bdc1e58184"
server_key_version: "encryptionkey@0"
I20260619 02:25:03.650842 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "2556aa76b8a442ef84a5ed3fad5f7189"
format_stamp: "Formatted at 2026-06-19 02:25:03 on dist-test-slave-xkrt"
server_key: "85e5eb4b308a21300904b80a0cfaaec9"
server_key_iv: "7c12580c2da3558e71a919bdc1e58184"
server_key_version: "encryptionkey@0"
I20260619 02:25:03.654366 5724 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:25:03.656637 5738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:03.657884 5724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:03.658017 5724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "2556aa76b8a442ef84a5ed3fad5f7189"
format_stamp: "Formatted at 2026-06-19 02:25:03 on dist-test-slave-xkrt"
server_key: "85e5eb4b308a21300904b80a0cfaaec9"
server_key_iv: "7c12580c2da3558e71a919bdc1e58184"
server_key_version: "encryptionkey@0"
I20260619 02:25:03.658128 5724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:03.680583 5724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:03.681247 5724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:03.681453 5724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:03.688637 5724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42393
I20260619 02:25:03.688644 5790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42393 every 8 connection(s)
I20260619 02:25:03.689574 5724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:25:03.692653 5791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:03.697970 5791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189: Bootstrap starting.
I20260619 02:25:03.699283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 5724
I20260619 02:25:03.699402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:25:03.699687 392 external_mini_cluster.cc:1428] Setting key afcfc1611aa00b1a232e922026d084e3
I20260619 02:25:03.700680 5791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:03.701444 5791 log.cc:824] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:03.703578 5791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189: No bootstrap required, opened a new log
I20260619 02:25:03.707667 5791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2556aa76b8a442ef84a5ed3fad5f7189" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42393 } }
I20260619 02:25:03.707960 5791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:03.708027 5791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2556aa76b8a442ef84a5ed3fad5f7189, State: Initialized, Role: FOLLOWER
I20260619 02:25:03.708437 5791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2556aa76b8a442ef84a5ed3fad5f7189" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42393 } }
I20260619 02:25:03.708563 5791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:25:03.708650 5791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:25:03.708771 5791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:03.709105 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:03.700852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.0.98.62:42393)
0619 02:25:03.701461 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:03.701475 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:03.701508 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:25:03.702403 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:03.702422 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:03.702735 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:03.703018 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:03.703358 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.704363 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.705633 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.706162 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.706284 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:03.707167 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:03.707172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:03.707183 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:03.707565 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:03.707617 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:03.707635 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:03.707711 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:03.707908 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 02:25:03.708211 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":131,"threads_started":1}
I20260619 02:25:03.709792 5791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2556aa76b8a442ef84a5ed3fad5f7189" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42393 } }
I20260619 02:25:03.710160 5791 leader_election.cc:302] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2556aa76b8a442ef84a5ed3fad5f7189; no voters:
I20260619 02:25:03.710474 5791 leader_election.cc:288] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:25:03.710597 5796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:03.710855 5796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [term 1 LEADER]: Becoming Leader. State: Replica: 2556aa76b8a442ef84a5ed3fad5f7189, State: Running, Role: LEADER
I20260619 02:25:03.711160 5796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2556aa76b8a442ef84a5ed3fad5f7189" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42393 } }
I20260619 02:25:03.711689 5791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:03.712131 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:25:03.712858 5798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2556aa76b8a442ef84a5ed3fad5f7189. Latest consensus state: current_term: 1 leader_uuid: "2556aa76b8a442ef84a5ed3fad5f7189" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2556aa76b8a442ef84a5ed3fad5f7189" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42393 } } }
I20260619 02:25:03.712982 5798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:03.714125 5805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:03.713310 5797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2556aa76b8a442ef84a5ed3fad5f7189" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2556aa76b8a442ef84a5ed3fad5f7189" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42393 } } }
I20260619 02:25:03.714232 5797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:03.716739 5805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:03.722016 5805 catalog_manager.cc:1347] Generated new cluster ID: 7f88eaccf01747ddb3a4ee538e7a4443
I20260619 02:25:03.722084 5805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:03.740885 5805 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:25:03.742180 5805 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:03.749647 5805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2556aa76b8a442ef84a5ed3fad5f7189: Generated new TSK 0
I20260619 02:25:03.750470 5805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:25:03.764823 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:03.759569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.0.98.62:42393)
0619 02:25:03.759699 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:03.759704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:03.759886 (+ 182us) server_negotiation.cc:407] Connection header received
0619 02:25:03.760013 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:03.760016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:03.760072 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:03.760187 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:25:03.760380 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.761197 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.762193 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.762574 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.762698 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:03.762973 (+ 275us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:25:03.764461 (+ 1488us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37673/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 02:25:03.764687 (+ 226us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56248: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37673/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 02:25:03.764892 5794 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56248: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37673/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 02:25:03.766635 5820 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:03.759504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42393 (local address 127.0.0.1:56248)
0619 02:25:03.759802 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:03.759814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:03.759905 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:03.760189 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:03.760191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:03.760196 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:25:03.760286 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:03.760290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.761306 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.761308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:03.762086 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:03.762095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.762709 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.762724 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:03.762788 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:03.762809 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:25:03.764626 (+ 1817us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37673/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 02:25:03.766470 (+ 1844us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42393: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37673/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":1578,"thread_start_us":121,"threads_started":1}
I20260619 02:25:03.767256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 5724
E20260619 02:25:03.775832 5821 webserver.cc:533] Webserver: sslize failed:
E20260619 02:25:03.775894 5821 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T02:25:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (301 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T02:25:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:03Z Disabled control of system clock
I20260619 02:25:03.813797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44243
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44243
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/testchainca.cert with env {}
W20260619 02:25:03.923707 5825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:03.923981 5825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:03.924022 5825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:03.928181 5825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:03.928254 5825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:03.928270 5825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:03.928287 5825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:03.928303 5825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:25:03.928320 5825 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:25:03.928334 5825 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 02:25:03.933851 5825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44243
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:03.934847 5825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:03.935801 5825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:03.944514 5831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:03.944578 5825 server_base.cc:1034] running on GCE node
W20260619 02:25:03.944525 5830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:03.944520 5833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:03.945441 5825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:03.946563 5825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:03.947758 5825 hybrid_clock.cc:648] HybridClock initialized: now 1781835903947742 us; error 32 us; skew 500 ppm
I20260619 02:25:03.949815 5825 webserver.cc:460] Webserver started at http://127.0.98.62:43187/ using document root <none> and password file <none>
I20260619 02:25:03.950398 5825 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:03.950490 5825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:03.950717 5825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:03.952732 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "e9a8f949881e4df8a80936e5d9945b7b"
format_stamp: "Formatted at 2026-06-19 02:25:03 on dist-test-slave-xkrt"
server_key: "7381c680ca61584896b6401f60ce342e"
server_key_iv: "ecc6d4e2febfd425a72b516db3b30885"
server_key_version: "encryptionkey@0"
I20260619 02:25:03.953313 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9a8f949881e4df8a80936e5d9945b7b"
format_stamp: "Formatted at 2026-06-19 02:25:03 on dist-test-slave-xkrt"
server_key: "7381c680ca61584896b6401f60ce342e"
server_key_iv: "ecc6d4e2febfd425a72b516db3b30885"
server_key_version: "encryptionkey@0"
I20260619 02:25:03.956990 5825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:25:03.959506 5839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:03.960731 5825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:03.960971 5825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "e9a8f949881e4df8a80936e5d9945b7b"
format_stamp: "Formatted at 2026-06-19 02:25:03 on dist-test-slave-xkrt"
server_key: "7381c680ca61584896b6401f60ce342e"
server_key_iv: "ecc6d4e2febfd425a72b516db3b30885"
server_key_version: "encryptionkey@0"
I20260619 02:25:03.961090 5825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:03.979939 5825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:03.980634 5825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:03.980830 5825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:03.987569 5825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44243
I20260619 02:25:03.987592 5891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44243 every 8 connection(s)
I20260619 02:25:03.988469 5825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:25:03.989218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 5825
I20260619 02:25:03.989348 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:25:03.989706 392 external_mini_cluster.cc:1428] Setting key 59abecaae04b7262bc9c6a354ae41e04
I20260619 02:25:03.992089 5892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:03.997337 5892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b: Bootstrap starting.
I20260619 02:25:03.998790 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:03.991146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.0.98.62:44243)
0619 02:25:03.991833 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:03.991847 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:03.991889 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:25:03.992586 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:03.992604 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:03.992983 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:03.993327 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:03.993753 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.994741 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.995621 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:03.996073 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:03.996246 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:03.997081 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:03.997086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:03.997097 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:03.997513 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:03.997627 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:03.997649 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:03.997750 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:03.997879 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 02:25:03.998124 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":193,"threads_started":1}
I20260619 02:25:03.999624 5892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:04.000416 5892 log.cc:824] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:04.001581 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:25:04.002694 5892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b: No bootstrap required, opened a new log
I20260619 02:25:04.006386 5892 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a8f949881e4df8a80936e5d9945b7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } }
I20260619 02:25:04.006594 5892 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:04.006664 5892 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a8f949881e4df8a80936e5d9945b7b, State: Initialized, Role: FOLLOWER
I20260619 02:25:04.007071 5892 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a8f949881e4df8a80936e5d9945b7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } }
I20260619 02:25:04.007210 5892 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:25:04.007275 5892 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:25:04.007367 5892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:04.008474 5892 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a8f949881e4df8a80936e5d9945b7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } }
I20260619 02:25:04.008800 5892 leader_election.cc:302] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9a8f949881e4df8a80936e5d9945b7b; no voters:
I20260619 02:25:04.009104 5892 leader_election.cc:288] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:25:04.009231 5897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:04.009472 5897 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [term 1 LEADER]: Becoming Leader. State: Replica: e9a8f949881e4df8a80936e5d9945b7b, State: Running, Role: LEADER
I20260619 02:25:04.009778 5897 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a8f949881e4df8a80936e5d9945b7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } }
I20260619 02:25:04.010231 5892 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:04.011456 5899 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9a8f949881e4df8a80936e5d9945b7b. Latest consensus state: current_term: 1 leader_uuid: "e9a8f949881e4df8a80936e5d9945b7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a8f949881e4df8a80936e5d9945b7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } } }
I20260619 02:25:04.011502 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9a8f949881e4df8a80936e5d9945b7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a8f949881e4df8a80936e5d9945b7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } } }
I20260619 02:25:04.011575 5899 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:04.011592 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:04.011951 5906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:04.014981 5906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:04.020294 5906 catalog_manager.cc:1347] Generated new cluster ID: fbd579026e9d44d1ad3eca1a8a412a78
I20260619 02:25:04.020449 5906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:04.033756 5906 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:25:04.034665 5906 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:04.040196 5906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9a8f949881e4df8a80936e5d9945b7b: Generated new TSK 0
I20260619 02:25:04.040874 5906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:25:04.119194 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:04.055041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34388 (local address 127.0.98.62:44243)
0619 02:25:04.055165 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:04.055170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:04.055338 (+ 168us) server_negotiation.cc:407] Connection header received
0619 02:25:04.055579 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:04.055581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:04.055626 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:04.055737 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:25:04.055978 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:04.056497 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:04.057362 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:04.057633 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:04.057744 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:04.058068 (+ 324us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:25:04.118681 (+ 60613us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 02:25:04.119038 (+ 357us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34388: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 02:25:04.119287 5895 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34388: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 02:25:04.119295 5921 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:04.054933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44243 (local address 127.0.0.1:34388)
0619 02:25:04.055246 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:04.055259 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:04.055439 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:04.055737 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:04.055740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:04.055748 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:25:04.055864 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:04.055878 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:04.056610 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:04.056614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:04.057242 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:04.057249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:04.057743 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:04.057764 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:04.057864 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:04.057886 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:25:04.118940 (+ 61054us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 02:25:04.119119 (+ 179us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44243: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":111,"threads_started":1}
I20260619 02:25:04.120237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 5825
2026-06-19T02:25:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (333 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (123 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:25:04 dist-test-slave-xkrt krb5kdc[5933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:25:04 dist-test-slave-xkrt krb5kdc[5933](info): set up 2 sockets
Jun 19 02:25:04 dist-test-slave-xkrt krb5kdc[5933](info): commencing operation
krb5kdc: starting...
W20260619 02:25:05.417897 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:25:05 dist-test-slave-xkrt krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:25:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:25:05.536180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44077
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:05.645185 5946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:05.645437 5946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:05.645475 5946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:05.649448 5946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:05.649508 5946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:05.649533 5946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:05.649566 5946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:05.649582 5946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:25:05.654793 5946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:05.655723 5946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:05.656684 5946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:05.665004 5954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:05.665040 5946 server_base.cc:1034] running on GCE node
W20260619 02:25:05.665006 5951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:05.665005 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:05.665850 5946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:05.666908 5946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:05.668092 5946 hybrid_clock.cc:648] HybridClock initialized: now 1781835905668064 us; error 40 us; skew 500 ppm
Jun 19 02:25:05 dist-test-slave-xkrt krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:05.670845 5946 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:25:05.671693 5946 webserver.cc:460] Webserver started at http://127.0.98.62:39373/ using document root <none> and password file <none>
I20260619 02:25:05.672236 5946 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:05.672329 5946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:05.672581 5946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:05.674644 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "bf11c7754d5d4cba9babd59cc20376c6"
format_stamp: "Formatted at 2026-06-19 02:25:05 on dist-test-slave-xkrt"
server_key: "5c3b400d585fe4ee11ae840c29126286"
server_key_iv: "28c679ea8ad156f0e85c224796af49c3"
server_key_version: "encryptionkey@0"
I20260619 02:25:05.675209 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf11c7754d5d4cba9babd59cc20376c6"
format_stamp: "Formatted at 2026-06-19 02:25:05 on dist-test-slave-xkrt"
server_key: "5c3b400d585fe4ee11ae840c29126286"
server_key_iv: "28c679ea8ad156f0e85c224796af49c3"
server_key_version: "encryptionkey@0"
I20260619 02:25:05.678967 5946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:25:05.681492 5961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:05.682775 5946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:25:05.682927 5946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "bf11c7754d5d4cba9babd59cc20376c6"
format_stamp: "Formatted at 2026-06-19 02:25:05 on dist-test-slave-xkrt"
server_key: "5c3b400d585fe4ee11ae840c29126286"
server_key_iv: "28c679ea8ad156f0e85c224796af49c3"
server_key_version: "encryptionkey@0"
I20260619 02:25:05.683041 5946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:05.696894 5946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:05.700103 5946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:05.700310 5946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:05.706976 5946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44077
I20260619 02:25:05.707038 6013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44077 every 8 connection(s)
I20260619 02:25:05.707913 5946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:25:05.710635 6014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:05.712080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 5946
I20260619 02:25:05.712221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:25:05.712548 392 external_mini_cluster.cc:1428] Setting key 76116a277275cec43b84ae26033848ac
I20260619 02:25:05.717227 6014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6: Bootstrap starting.
I20260619 02:25:05.719797 6014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:05.720675 6014 log.cc:824] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:25:05 dist-test-slave-xkrt krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:25:05.722779 6014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6: No bootstrap required, opened a new log
I20260619 02:25:05.727414 6014 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf11c7754d5d4cba9babd59cc20376c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260619 02:25:05.727671 6014 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:05.727718 6014 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf11c7754d5d4cba9babd59cc20376c6, State: Initialized, Role: FOLLOWER
I20260619 02:25:05.727793 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:05.714059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.0.98.62:44077)
0619 02:25:05.714472 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:05.714490 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:05.714532 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:25:05.715369 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:05.715392 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:05.715797 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:05.716170 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:05.716916 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:05.718372 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:05.719228 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:05.719891 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:05.720075 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:05.722755 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:05.722761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:05.722775 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:05.722806 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:05.724714 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:05.725148 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:05.725156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:05.725165 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:05.725325 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:05.725688 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:05.725692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:05.725694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:05.726170 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:05.726374 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:05.726559 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 02:25:05.726850 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260619 02:25:05.728173 6014 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf11c7754d5d4cba9babd59cc20376c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260619 02:25:05.728327 6014 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:25:05.728379 6014 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:25:05.728477 6014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:05.729386 6014 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf11c7754d5d4cba9babd59cc20376c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260619 02:25:05.729753 6014 leader_election.cc:302] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf11c7754d5d4cba9babd59cc20376c6; no voters:
I20260619 02:25:05.730053 6014 leader_election.cc:288] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:25:05.730229 6019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:05.730500 6019 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [term 1 LEADER]: Becoming Leader. State: Replica: bf11c7754d5d4cba9babd59cc20376c6, State: Running, Role: LEADER
I20260619 02:25:05.730794 6019 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf11c7754d5d4cba9babd59cc20376c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260619 02:25:05.731266 6014 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:05.732692 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf11c7754d5d4cba9babd59cc20376c6. Latest consensus state: current_term: 1 leader_uuid: "bf11c7754d5d4cba9babd59cc20376c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf11c7754d5d4cba9babd59cc20376c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } } }
I20260619 02:25:05.732681 6020 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf11c7754d5d4cba9babd59cc20376c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf11c7754d5d4cba9babd59cc20376c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } } }
I20260619 02:25:05.732854 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:05.732854 6020 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:05.733279 6031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:05.736654 6031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:05.742640 6031 catalog_manager.cc:1347] Generated new cluster ID: cd9c4778e1394e5e8913145aabcafcca
I20260619 02:25:05.742736 6031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:05.765692 6031 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:25:05.766832 6031 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:05.772542 6031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf11c7754d5d4cba9babd59cc20376c6: Generated new TSK 0
I20260619 02:25:05.773268 6031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:25:05.809091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44077
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:05.920693 6040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:05.920958 6040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:05.921002 6040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:05.925179 6040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:05.925239 6040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:05.925343 6040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:25:05.930619 6040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:05.931734 6040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:05.932884 6040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:05.943032 6046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:05.943058 6048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:05.943035 6045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:05.943480 6040 server_base.cc:1034] running on GCE node
I20260619 02:25:05.943914 6040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:05.944626 6040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:05.945791 6040 hybrid_clock.cc:648] HybridClock initialized: now 1781835905945771 us; error 29 us; skew 500 ppm
Jun 19 02:25:05 dist-test-slave-xkrt krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:05.953485 6040 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:25:05.954403 6040 webserver.cc:460] Webserver started at http://127.0.98.1:40289/ using document root <none> and password file <none>
I20260619 02:25:05.954926 6040 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:05.954985 6040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:05.955147 6040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:05.957149 6040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "aed47ed8d560493bb0f5d49bb4bdeea7"
format_stamp: "Formatted at 2026-06-19 02:25:05 on dist-test-slave-xkrt"
server_key: "2446f28fb7a224fea91b5c453b3dcd99"
server_key_iv: "89fd69b5a40948ab6adfd1e90e91d6b7"
server_key_version: "encryptionkey@0"
I20260619 02:25:05.957724 6040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aed47ed8d560493bb0f5d49bb4bdeea7"
format_stamp: "Formatted at 2026-06-19 02:25:05 on dist-test-slave-xkrt"
server_key: "2446f28fb7a224fea91b5c453b3dcd99"
server_key_iv: "89fd69b5a40948ab6adfd1e90e91d6b7"
server_key_version: "encryptionkey@0"
I20260619 02:25:05.961263 6040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:25:05.963722 6055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:05.964939 6040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:05.965087 6040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "aed47ed8d560493bb0f5d49bb4bdeea7"
format_stamp: "Formatted at 2026-06-19 02:25:05 on dist-test-slave-xkrt"
server_key: "2446f28fb7a224fea91b5c453b3dcd99"
server_key_iv: "89fd69b5a40948ab6adfd1e90e91d6b7"
server_key_version: "encryptionkey@0"
I20260619 02:25:05.965201 6040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:05.988634 6040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:05.991562 6040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:05.991763 6040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:05.992365 6040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:05.993181 6040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:05.993252 6040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:05.993323 6040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:05.993386 6040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:06.002175 6040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41153
I20260619 02:25:06.002199 6168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41153 every 8 connection(s)
I20260619 02:25:06.003104 6040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:25:06.005311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 6040
I20260619 02:25:06.005419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:25:06.005744 392 external_mini_cluster.cc:1428] Setting key 0e6cd8a59d880ed48331766f1117e7b3
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 02:25:06.017666 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:06.004950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56691 (local address 127.0.98.62:44077)
0619 02:25:06.005130 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:06.005137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:06.006295 (+ 1158us) server_negotiation.cc:407] Connection header received
0619 02:25:06.007681 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:06.007687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:06.007750 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:06.007842 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:06.009129 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.009764 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.010921 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.011157 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.011268 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:06.013817 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:06.013821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:06.013824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:06.013845 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:06.015360 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:06.015901 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:06.015906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:06.015908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:06.015988 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:06.016401 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:06.016403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:06.016405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:06.016607 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:06.016730 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:06.017295 (+ 565us) server_negotiation.cc:299] Negotiation successful
0619 02:25:06.017437 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:25:06.018421 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:06.005329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:56691)
0619 02:25:06.005943 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:06.005995 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:06.007464 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:06.008056 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:06.008069 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:06.008439 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:06.008928 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:06.008942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.009893 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.009896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:06.010794 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:06.010804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.011314 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.011318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:06.011494 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:06.012143 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:06.012159 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:06.013654 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:06.015486 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:06.015495 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:06.015512 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:06.015785 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:06.016088 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:06.016091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:06.016093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:06.016227 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:06.016732 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:06.016739 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:06.017050 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 02:25:06.017243 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 02:25:06.017587 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":182,"threads_started":1}
I20260619 02:25:06.019737 6169 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44077
I20260619 02:25:06.019980 6169 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:06.020464 6169 heartbeater.cc:511] Master 127.0.98.62:44077 requested a full tablet report, sending...
I20260619 02:25:06.022212 5978 ts_manager.cc:194] Registered new tserver with Master: aed47ed8d560493bb0f5d49bb4bdeea7 (127.0.98.1:41153)
I20260619 02:25:06.023891 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:25:06.033401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44077
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:06.143092 6174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:06.143358 6174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:06.143402 6174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:06.147400 6174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:06.147466 6174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:06.147562 6174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:25:06.152853 6174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:06.153937 6174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:06.155011 6174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:06.164355 6180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:06.164366 6182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:06.164391 6179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:06.164664 6174 server_base.cc:1034] running on GCE node
I20260619 02:25:06.165060 6174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:06.165771 6174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:06.166927 6174 hybrid_clock.cc:648] HybridClock initialized: now 1781835906166913 us; error 27 us; skew 500 ppm
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:06.170089 6174 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:25:06.170912 6174 webserver.cc:460] Webserver started at http://127.0.98.2:36231/ using document root <none> and password file <none>
I20260619 02:25:06.171466 6174 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:06.171526 6174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:06.171756 6174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:06.173784 6174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "c22bda73e64d4140acce7db58dc7dd8c"
format_stamp: "Formatted at 2026-06-19 02:25:06 on dist-test-slave-xkrt"
server_key: "beebd1bd1deaf42dca245dcf0af41d10"
server_key_iv: "fcbdc3184ddf76c3db39d8b689a77f9e"
server_key_version: "encryptionkey@0"
I20260619 02:25:06.174336 6174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c22bda73e64d4140acce7db58dc7dd8c"
format_stamp: "Formatted at 2026-06-19 02:25:06 on dist-test-slave-xkrt"
server_key: "beebd1bd1deaf42dca245dcf0af41d10"
server_key_iv: "fcbdc3184ddf76c3db39d8b689a77f9e"
server_key_version: "encryptionkey@0"
I20260619 02:25:06.178052 6174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:25:06.180465 6189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:06.181823 6174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:06.181945 6174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "c22bda73e64d4140acce7db58dc7dd8c"
format_stamp: "Formatted at 2026-06-19 02:25:06 on dist-test-slave-xkrt"
server_key: "beebd1bd1deaf42dca245dcf0af41d10"
server_key_iv: "fcbdc3184ddf76c3db39d8b689a77f9e"
server_key_version: "encryptionkey@0"
I20260619 02:25:06.182071 6174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:06.208580 6174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:06.212085 6174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:06.212285 6174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:06.212898 6174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:06.213812 6174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:06.213861 6174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:06.213922 6174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:06.213953 6174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:06.223031 6174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33683
I20260619 02:25:06.223055 6302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33683 every 8 connection(s)
I20260619 02:25:06.223923 6174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:25:06.229786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 6174
I20260619 02:25:06.229903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:25:06.230154 392 external_mini_cluster.cc:1428] Setting key 94c1fb9737c0de07e00e77e520de373a
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:25:06.239718 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:06.225835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49955 (local address 127.0.98.62:44077)
0619 02:25:06.225974 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:06.225978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:06.226803 (+ 825us) server_negotiation.cc:407] Connection header received
0619 02:25:06.228267 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:06.228270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:06.228320 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:06.228403 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:06.229516 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.230184 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.231710 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.231967 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.232101 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:06.235854 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:06.235857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:06.235859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:06.235883 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:06.237377 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:06.238050 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:06.238053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:06.238054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:06.238127 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:06.238525 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:06.238528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:06.238529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:06.238746 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:06.238860 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:06.239494 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 02:25:06.239595 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:25:06.240928 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:06.226229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:49955)
0619 02:25:06.226647 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:06.226688 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:06.228058 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:06.228561 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:06.228568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:06.228905 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:06.229375 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:06.229387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.230321 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.230324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:06.231542 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:06.231551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.232139 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.232143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:06.232341 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:06.233112 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:06.233137 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:06.235675 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:06.237500 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:06.237509 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:06.237529 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:06.237947 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:06.238220 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:06.238225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:06.238228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:06.238431 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:06.238874 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:06.238881 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:06.239218 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 02:25:06.239456 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 02:25:06.239811 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
I20260619 02:25:06.242128 6303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44077
I20260619 02:25:06.242375 6303 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:06.242869 6303 heartbeater.cc:511] Master 127.0.98.62:44077 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 02:25:06.243952 5978 ts_manager.cc:194] Registered new tserver with Master: c22bda73e64d4140acce7db58dc7dd8c (127.0.98.2:33683)
I20260619 02:25:06.244525 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:25:06.258800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44077
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:06.368468 6308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:06.368705 6308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:06.368747 6308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:06.372766 6308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:06.372821 6308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:06.372915 6308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:25:06.378151 6308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:06.379133 6308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:06.380105 6308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:06.389647 6313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:06.389987 6308 server_base.cc:1034] running on GCE node
W20260619 02:25:06.389655 6314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:06.389658 6316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:06.390410 6308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:06.391052 6308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:06.392244 6308 hybrid_clock.cc:648] HybridClock initialized: now 1781835906392231 us; error 23 us; skew 500 ppm
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:06.395177 6308 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:25:06.396029 6308 webserver.cc:460] Webserver started at http://127.0.98.3:37025/ using document root <none> and password file <none>
I20260619 02:25:06.396591 6308 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:06.396653 6308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:06.396863 6308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:06.398876 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "ee84a3bf19244471ad30e7d233f73c72"
format_stamp: "Formatted at 2026-06-19 02:25:06 on dist-test-slave-xkrt"
server_key: "5d0eca3aabd923d94c8936d8ab4606c8"
server_key_iv: "25c43699ba4417661704ccab5e29acde"
server_key_version: "encryptionkey@0"
I20260619 02:25:06.399442 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ee84a3bf19244471ad30e7d233f73c72"
format_stamp: "Formatted at 2026-06-19 02:25:06 on dist-test-slave-xkrt"
server_key: "5d0eca3aabd923d94c8936d8ab4606c8"
server_key_iv: "25c43699ba4417661704ccab5e29acde"
server_key_version: "encryptionkey@0"
I20260619 02:25:06.403034 6308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:25:06.405364 6323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:06.406586 6308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:06.406704 6308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "ee84a3bf19244471ad30e7d233f73c72"
format_stamp: "Formatted at 2026-06-19 02:25:06 on dist-test-slave-xkrt"
server_key: "5d0eca3aabd923d94c8936d8ab4606c8"
server_key_iv: "25c43699ba4417661704ccab5e29acde"
server_key_version: "encryptionkey@0"
I20260619 02:25:06.406805 6308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:06.422586 6308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:06.425802 6308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:06.426000 6308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:06.426563 6308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:06.427327 6308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:06.427371 6308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:06.427444 6308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:06.427485 6308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:06.436205 6308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39331
I20260619 02:25:06.436232 6436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39331 every 8 connection(s)
I20260619 02:25:06.437076 6308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:25:06.444720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 6308
I20260619 02:25:06.444823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:25:06.445416 392 external_mini_cluster.cc:1428] Setting key 7724e01081f309f366a31cf2816c2ce2
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:25:06.450740 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:06.438769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52775 (local address 127.0.98.62:44077)
0619 02:25:06.438900 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:06.438904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:06.439694 (+ 790us) server_negotiation.cc:407] Connection header received
0619 02:25:06.441053 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:06.441057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:06.441119 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:06.441194 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:06.442397 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.442930 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.444071 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.444305 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.444419 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:06.447250 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:06.447253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:06.447255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:06.447277 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:06.448605 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:06.449135 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:06.449138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:06.449139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:06.449224 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:06.449732 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:06.449735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:06.449736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:06.449949 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:06.450037 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:06.450504 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 02:25:06.450612 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:25:06.451534 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:06.439074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:52775)
0619 02:25:06.439552 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:06.439592 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:06.440830 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:06.441360 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:06.441367 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:06.441777 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:06.442243 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:06.442255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.443056 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.443059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:06.443931 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:06.443940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.444456 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.444460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:06.444631 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:06.445217 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:06.445234 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:06.447086 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:06.448728 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:06.448732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:06.448744 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:06.449016 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:06.449335 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:06.449338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:06.449340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:06.449483 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:06.450045 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:06.450050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:06.450278 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 02:25:06.450427 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 02:25:06.450693 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":122,"threads_started":1}
I20260619 02:25:06.452755 6437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44077
I20260619 02:25:06.452991 6437 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:06.453462 6437 heartbeater.cc:511] Master 127.0.98.62:44077 requested a full tablet report, sending...
I20260619 02:25:06.454481 5978 ts_manager.cc:194] Registered new tserver with Master: ee84a3bf19244471ad30e7d233f73c72 (127.0.98.3:39331)
I20260619 02:25:06.455001 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52775
I20260619 02:25:06.459245 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/test-admin.keytab.
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:06.475474 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 02:25:06.478528 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:06.476933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.0.98.1:41153)
0619 02:25:06.477212 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:06.477226 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:06.477250 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:25:06.477303 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:06.477308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:06.477521 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:06.477710 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:06.478293 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44316: BlockingRecv error: recv got EOF from 127.0.0.1:44316 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
W20260619 02:25:06.478760 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/krb5cc)
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:06.479902 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:06.480890 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 02:25:06.491184 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:06.481830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.0.98.1:41153)
0619 02:25:06.481952 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:06.481958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:06.482071 (+ 113us) server_negotiation.cc:407] Connection header received
0619 02:25:06.482166 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:06.482168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:06.482205 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:06.482272 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:06.482824 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.483644 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.484530 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.485075 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.485159 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:06.487871 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:06.487880 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:06.487892 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:06.487916 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:06.489610 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:06.490017 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:06.490022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:06.490023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:06.490104 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:06.490340 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:06.490343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:06.490344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:06.490601 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:06.490762 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:06.490916 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 02:25:06.491041 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 02:25:06.495102 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:06.494041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.0.98.1:41153)
0619 02:25:06.494152 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:06.494156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:06.494299 (+ 143us) server_negotiation.cc:407] Connection header received
0619 02:25:06.494445 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:06.494448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:06.494492 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:06.494574 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:06.494979 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44340: BlockingRecv error: recv got EOF from 127.0.0.1:44340 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 02:25:06.495277 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/krb5cc)
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:06.496332 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:06.497341 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 02:25:06.506012 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:06.498301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44348 (local address 127.0.98.1:41153)
0619 02:25:06.498438 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:06.498442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:06.498546 (+ 104us) server_negotiation.cc:407] Connection header received
0619 02:25:06.498683 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:06.498686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:06.498731 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:06.498820 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:06.499292 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.499822 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.500598 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.500835 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.500944 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:06.503169 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:06.503172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:06.503175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:06.503203 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:06.504477 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:06.504877 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:06.504881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:06.504883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:06.504960 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:06.505228 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:06.505231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:06.505233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:06.505387 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:06.505518 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:06.505760 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 02:25:06.505876 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 02:25:06.508775 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:06.507610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.0.98.1:41153)
0619 02:25:06.507719 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:06.507723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:06.507828 (+ 105us) server_negotiation.cc:407] Connection header received
0619 02:25:06.507912 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:06.507914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:06.507944 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:06.508003 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:06.508663 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44358: BlockingRecv error: recv got EOF from 127.0.0.1:44358 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 02:25:06.508862 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781835873883564-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:06.509927 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:06.510954 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 02:25:06.519312 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:06.511851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.0.98.1:41153)
0619 02:25:06.511990 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:06.511994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:06.512069 (+ 75us) server_negotiation.cc:407] Connection header received
0619 02:25:06.512158 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:06.512160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:06.512191 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:06.512254 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:06.512751 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.513221 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.514113 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:06.514312 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:06.514389 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:06.516646 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:06.516649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:06.516651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:06.516671 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:06.517938 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:06.518312 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:06.518314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:06.518315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:06.518382 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:06.518615 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:06.518617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:06.518619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:06.518752 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:06.518855 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:06.519034 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 02:25:06.519112 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:25:06.520236 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 6040
I20260619 02:25:06.525866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 6174
I20260619 02:25:06.531873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 6308
I20260619 02:25:06.538686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 5946
2026-06-19T02:25:06Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2177 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[6457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[6457](info): set up 2 sockets
Jun 19 02:25:06 dist-test-slave-xkrt krb5kdc[6457](info): commencing operation
krb5kdc: starting...
W20260619 02:25:07.594300 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:25:07 dist-test-slave-xkrt krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:25:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 02:25:07.716022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:07.822769 6470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:07.823032 6470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:07.823071 6470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:07.827162 6470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:07.827229 6470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:07.827248 6470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:07.827265 6470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:07.827286 6470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:25:07.832557 6470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:07.833729 6470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:07.834751 6470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:07.843407 6475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:07.843448 6476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:07.843495 6470 server_base.cc:1034] running on GCE node
W20260619 02:25:07.843448 6478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:07.844197 6470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:07.845309 6470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:07.846513 6470 hybrid_clock.cc:648] HybridClock initialized: now 1781835907846507 us; error 58 us; skew 500 ppm
Jun 19 02:25:07 dist-test-slave-xkrt krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835907, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:07.849359 6470 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 02:25:07.850303 6470 webserver.cc:460] Webserver started at http://127.0.98.62:36623/ using document root <none> and password file <none>
I20260619 02:25:07.850807 6470 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:07.850905 6470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:07.851107 6470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:07.853096 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "e8e16d7c823b4c189a05e3c8bb416c35"
format_stamp: "Formatted at 2026-06-19 02:25:07 on dist-test-slave-xkrt"
server_key: "8f07fed997dec8d9a96f023600ffb24c"
server_key_iv: "1a5821d5a34a1063801d43448e43fdb7"
server_key_version: "encryptionkey@0"
I20260619 02:25:07.853675 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "e8e16d7c823b4c189a05e3c8bb416c35"
format_stamp: "Formatted at 2026-06-19 02:25:07 on dist-test-slave-xkrt"
server_key: "8f07fed997dec8d9a96f023600ffb24c"
server_key_iv: "1a5821d5a34a1063801d43448e43fdb7"
server_key_version: "encryptionkey@0"
I20260619 02:25:07.857363 6470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 02:25:07.859807 6485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:07.861125 6470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:25:07.861253 6470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "e8e16d7c823b4c189a05e3c8bb416c35"
format_stamp: "Formatted at 2026-06-19 02:25:07 on dist-test-slave-xkrt"
server_key: "8f07fed997dec8d9a96f023600ffb24c"
server_key_iv: "1a5821d5a34a1063801d43448e43fdb7"
server_key_version: "encryptionkey@0"
I20260619 02:25:07.861403 6470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:07.880267 6470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:07.883699 6470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:07.883898 6470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:07.890800 6470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40883
I20260619 02:25:07.891085 6537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40883 every 8 connection(s)
I20260619 02:25:07.891813 6470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:25:07.894557 6538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:07.899969 6538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35: Bootstrap starting.
I20260619 02:25:07.901831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 6470
I20260619 02:25:07.901962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:25:07.902235 392 external_mini_cluster.cc:1428] Setting key a52dd4f3bdf4e2f38345281c2ad59866
I20260619 02:25:07.902557 6538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:07.903451 6538 log.cc:824] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:07.905375 6538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35: No bootstrap required, opened a new log
I20260619 02:25:07.909461 6538 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } }
I20260619 02:25:07.909776 6538 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:07.909850 6538 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e16d7c823b4c189a05e3c8bb416c35, State: Initialized, Role: FOLLOWER
I20260619 02:25:07.910372 6538 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } }
I20260619 02:25:07.910553 6538 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:25:07.910622 6538 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:25:07.910720 6538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 02:25:07 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 02:25:07.911690 6538 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } }
I20260619 02:25:07.912034 6538 leader_election.cc:302] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8e16d7c823b4c189a05e3c8bb416c35; no voters:
I20260619 02:25:07.912371 6538 leader_election.cc:288] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:25:07.912441 6543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:07.912701 6543 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [term 1 LEADER]: Becoming Leader. State: Replica: e8e16d7c823b4c189a05e3c8bb416c35, State: Running, Role: LEADER
I20260619 02:25:07.913050 6543 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } }
I20260619 02:25:07.913751 6538 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:07.914283 6543 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } }
I20260619 02:25:07.914482 6543 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:07.914690 6544 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8e16d7c823b4c189a05e3c8bb416c35. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } }
I20260619 02:25:07.914849 6544 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:07.916246 6553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:07.918298 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:07.903364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.0.98.62:40883)
0619 02:25:07.903836 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:07.903850 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:07.903883 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:25:07.904505 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:07.904521 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:07.904856 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:07.905122 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:07.906074 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:07.907048 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:07.908293 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:07.908757 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:07.908896 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:07.911754 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:07.911760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:07.911777 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:07.911804 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:07.914089 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:07.915447 (+ 1358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:07.915452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:07.915459 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:07.915629 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:07.915970 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:07.915973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:07.915976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:07.916474 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:07.916684 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:07.916925 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 02:25:07.917351 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":168,"threads_started":1}
I20260619 02:25:07.919130 6553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:07.925186 6553 catalog_manager.cc:1347] Generated new cluster ID: 9d73f6d34c9c4fa8aeb90a06b2f1baa6
I20260619 02:25:07.925279 6553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:07.960917 6553 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:25:07.961956 6553 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:07.965983 6553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35: Generated new TSK 0
I20260619 02:25:07.966851 6553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 02:25:07.997026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:08.106074 6564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:08.106375 6564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:08.106465 6564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:08.110440 6564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:08.110538 6564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:08.110625 6564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:25:08.116072 6564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:08.117133 6564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:08.118155 6564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:08.127732 6570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:08.127732 6572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:08.127744 6569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:08.127961 6564 server_base.cc:1034] running on GCE node
I20260619 02:25:08.128404 6564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:08.129061 6564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:08.130244 6564 hybrid_clock.cc:648] HybridClock initialized: now 1781835908130204 us; error 58 us; skew 500 ppm
Jun 19 02:25:08 dist-test-slave-xkrt krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:08.133105 6564 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 02:25:08.134066 6564 webserver.cc:460] Webserver started at http://127.0.98.1:39305/ using document root <none> and password file <none>
I20260619 02:25:08.134630 6564 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:08.134721 6564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:08.134956 6564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:08.136976 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "990e7d22b7cd4a0baefea77792def123"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "79d0ca1cae75a2f4a922b6cbe4f5cba0"
server_key_iv: "fd8e5ace6afc20d3369e34f659ad635e"
server_key_version: "encryptionkey@0"
I20260619 02:25:08.137616 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "990e7d22b7cd4a0baefea77792def123"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "79d0ca1cae75a2f4a922b6cbe4f5cba0"
server_key_iv: "fd8e5ace6afc20d3369e34f659ad635e"
server_key_version: "encryptionkey@0"
I20260619 02:25:08.141312 6564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:25:08.143782 6579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:08.145047 6564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:08.145187 6564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "990e7d22b7cd4a0baefea77792def123"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "79d0ca1cae75a2f4a922b6cbe4f5cba0"
server_key_iv: "fd8e5ace6afc20d3369e34f659ad635e"
server_key_version: "encryptionkey@0"
I20260619 02:25:08.145300 6564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:08.162016 6564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:08.165330 6564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:08.165617 6564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:08.166213 6564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:08.167019 6564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:08.167088 6564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:08.167158 6564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:08.167202 6564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:08.175936 6564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36963
I20260619 02:25:08.175971 6692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36963 every 8 connection(s)
I20260619 02:25:08.176801 6564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:25:08.183462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 6564
I20260619 02:25:08.183624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:25:08.183929 392 external_mini_cluster.cc:1428] Setting key 53fae036845f88de83089ce1cedfe18a
Jun 19 02:25:08 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 02:25:08.192000 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.178601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46509 (local address 127.0.98.62:40883)
0619 02:25:08.178757 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:08.178761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:08.179603 (+ 842us) server_negotiation.cc:407] Connection header received
0619 02:25:08.180941 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:08.180944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:08.180995 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:08.181083 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:08.182307 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.182918 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.184249 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.184559 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.184671 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.187704 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:08.187710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:08.187715 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:08.187750 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:08.189719 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.190243 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.190249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.190254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.190349 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.190695 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.190701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.190704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.190990 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:08.191113 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:08.191628 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 02:25:08.191781 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:25:08.192739 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.178950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.1:46509)
0619 02:25:08.179445 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:08.179489 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:08.180708 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:08.181256 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:08.181263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:08.181682 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:08.182156 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.182168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.183029 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.183031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.184134 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.184143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.184724 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.184728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.184935 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.185676 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:08.185692 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:08.187517 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:08.189844 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.189849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.189861 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.190130 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.190450 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.190452 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.190453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.190595 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.191122 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:08.191128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:08.191381 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 02:25:08.191545 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 02:25:08.191796 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:25:08.194000 6693 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40883
I20260619 02:25:08.194269 6693 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:08.194736 6693 heartbeater.cc:511] Master 127.0.98.62:40883 requested a full tablet report, sending...
I20260619 02:25:08.196308 6502 ts_manager.cc:194] Registered new tserver with Master: 990e7d22b7cd4a0baefea77792def123 (127.0.98.1:36963)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260619 02:25:08.197530 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 02:25:08.212358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:08.321128 6698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:08.321424 6698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:08.321498 6698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:08.325534 6698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:08.325640 6698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:08.325726 6698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:25:08.331007 6698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:08.332083 6698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:08.333106 6698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:08.342672 6703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:08.342784 6698 server_base.cc:1034] running on GCE node
W20260619 02:25:08.342675 6706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:08.342675 6704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:08.343433 6698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:08.344091 6698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:08.345290 6698 hybrid_clock.cc:648] HybridClock initialized: now 1781835908345261 us; error 47 us; skew 500 ppm
Jun 19 02:25:08 dist-test-slave-xkrt krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:08.348169 6698 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 02:25:08.349116 6698 webserver.cc:460] Webserver started at http://127.0.98.2:41203/ using document root <none> and password file <none>
I20260619 02:25:08.349743 6698 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:08.349831 6698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:08.350064 6698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:08.352147 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "12458f15b67c479ca476b132816f5ac5"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "72a13a32462f88f2c170d1c8bcd250d2"
server_key_iv: "37cde3b7c51502b5e83c5365f4268985"
server_key_version: "encryptionkey@0"
I20260619 02:25:08.352725 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "12458f15b67c479ca476b132816f5ac5"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "72a13a32462f88f2c170d1c8bcd250d2"
server_key_iv: "37cde3b7c51502b5e83c5365f4268985"
server_key_version: "encryptionkey@0"
I20260619 02:25:08.356504 6698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 02:25:08.359064 6713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:08.360549 6698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:25:08.360709 6698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "12458f15b67c479ca476b132816f5ac5"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "72a13a32462f88f2c170d1c8bcd250d2"
server_key_iv: "37cde3b7c51502b5e83c5365f4268985"
server_key_version: "encryptionkey@0"
I20260619 02:25:08.360857 6698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:08.371011 6698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:08.373989 6698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:08.374182 6698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:08.374732 6698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:08.375505 6698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:08.375574 6698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:08.375643 6698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:08.375694 6698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:08.384609 6698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36889
I20260619 02:25:08.384631 6826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36889 every 8 connection(s)
I20260619 02:25:08.385474 6698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:25:08.388324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 6698
I20260619 02:25:08.388463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:25:08.388773 392 external_mini_cluster.cc:1428] Setting key 588b10186c05a2d8eb5afbe296f87af8
Jun 19 02:25:08 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:25:08.399545 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.387285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42529 (local address 127.0.98.62:40883)
0619 02:25:08.387446 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:08.387451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:08.388250 (+ 799us) server_negotiation.cc:407] Connection header received
0619 02:25:08.389874 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:08.389877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:08.389936 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:08.390014 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:08.391116 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.391637 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.392751 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.392982 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.393093 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.395796 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:08.395800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:08.395802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:08.395824 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:08.397326 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.397951 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.397956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.397958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.398040 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.398394 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.398397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.398400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.398639 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:08.398745 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:08.399248 (+ 503us) server_negotiation.cc:299] Negotiation successful
0619 02:25:08.399382 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260619 02:25:08.400578 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.387668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.2:42529)
0619 02:25:08.388098 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:08.388140 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:08.389677 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:08.390172 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:08.390179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:08.390519 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:08.390978 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.390990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.391759 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.391764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.392628 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.392636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.393128 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.393132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.393291 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.393923 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:08.393939 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:08.395624 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:08.397509 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.397515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.397530 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.397833 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.398137 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.398141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.398143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.398284 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.398753 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:08.398761 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:08.399011 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 02:25:08.399196 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 02:25:08.399592 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260619 02:25:08.401873 6827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40883
I20260619 02:25:08.402115 6827 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:08.402634 6827 heartbeater.cc:511] Master 127.0.98.62:40883 requested a full tablet report, sending...
I20260619 02:25:08.403807 6502 ts_manager.cc:194] Registered new tserver with Master: 12458f15b67c479ca476b132816f5ac5 (127.0.98.2:36889)
I20260619 02:25:08.404676 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 02:25:08.416599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:08.526870 6832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:08.527140 6832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:08.527184 6832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:08.531260 6832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:08.531333 6832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:08.531394 6832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:25:08.536751 6832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:08.537889 6832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:08.538940 6832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:08.550532 6838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:08.550534 6837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:08.550544 6840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:08.550944 6832 server_base.cc:1034] running on GCE node
I20260619 02:25:08.551395 6832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:08.552047 6832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:08.553236 6832 hybrid_clock.cc:648] HybridClock initialized: now 1781835908553184 us; error 71 us; skew 500 ppm
Jun 19 02:25:08 dist-test-slave-xkrt krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:08.556078 6832 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 02:25:08.556936 6832 webserver.cc:460] Webserver started at http://127.0.98.3:35173/ using document root <none> and password file <none>
I20260619 02:25:08.557451 6832 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:08.557512 6832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:08.557753 6832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:08.559705 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "5ba57c2beb154a38adc35cdac706fcb1"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "d50d4a4f965d80a398b336e2205db7bf"
server_key_iv: "a6ce15a52d9388fad86bd973f7101c93"
server_key_version: "encryptionkey@0"
I20260619 02:25:08.560223 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5ba57c2beb154a38adc35cdac706fcb1"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "d50d4a4f965d80a398b336e2205db7bf"
server_key_iv: "a6ce15a52d9388fad86bd973f7101c93"
server_key_version: "encryptionkey@0"
I20260619 02:25:08.563885 6832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 02:25:08.566452 6847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:08.567652 6832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:08.567749 6832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "5ba57c2beb154a38adc35cdac706fcb1"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "d50d4a4f965d80a398b336e2205db7bf"
server_key_iv: "a6ce15a52d9388fad86bd973f7101c93"
server_key_version: "encryptionkey@0"
I20260619 02:25:08.567834 6832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:08.582049 6832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:08.585317 6832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:08.585469 6832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:08.586074 6832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:08.586844 6832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:08.586890 6832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:08.586930 6832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:08.586944 6832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:08.595772 6832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41937
I20260619 02:25:08.595809 6960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41937 every 8 connection(s)
I20260619 02:25:08.596671 6832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:25:08.602890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 6832
I20260619 02:25:08.603016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:25:08.603389 392 external_mini_cluster.cc:1428] Setting key ff276065bc77aa89b2991cc80a779d95
Jun 19 02:25:08 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 02:25:08.610478 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.598504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52717 (local address 127.0.98.62:40883)
0619 02:25:08.598639 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:08.598643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:08.599401 (+ 758us) server_negotiation.cc:407] Connection header received
0619 02:25:08.600677 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:08.600680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:08.600726 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:08.600803 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:08.602071 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.602598 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.604096 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.604426 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.604545 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.607142 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:08.607145 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:08.607148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:08.607174 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:08.608441 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.608940 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.608945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.608947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.609030 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.609359 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.609365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.609366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.609609 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:08.609716 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:08.610197 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 02:25:08.610309 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:25:08.611478 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.598805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.3:52717)
0619 02:25:08.599242 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:08.599285 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:08.600483 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:08.601005 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:08.601012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:08.601367 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:08.601944 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.601958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.602738 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.602741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.603995 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.604004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.604608 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.604613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.604822 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.605453 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:08.605492 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:08.606986 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:08.608565 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.608571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.608584 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.608836 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.609123 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.609126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.609128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.609264 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.609715 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:08.609721 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:08.609961 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 02:25:08.610159 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 02:25:08.610471 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":130,"threads_started":1}
I20260619 02:25:08.612644 6961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40883
I20260619 02:25:08.612887 6961 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:08.613400 6961 heartbeater.cc:511] Master 127.0.98.62:40883 requested a full tablet report, sending...
I20260619 02:25:08.614488 6502 ts_manager.cc:194] Registered new tserver with Master: 5ba57c2beb154a38adc35cdac706fcb1 (127.0.98.3:41937)
I20260619 02:25:08.615089 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52717
I20260619 02:25:08.617063 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 02:25:08.643909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36913
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40883,127.0.98.61:36913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:08.753055 6966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:08.753319 6966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:08.753358 6966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:08.757335 6966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:08.757392 6966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:08.757411 6966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:08.757428 6966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:08.757443 6966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:25:08.762807 6966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40883,127.0.98.61:36913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
W20260619 02:25:08.762986 6966 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:40883,127.0.98.61:36913'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 02:25:08.763844 6966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:08.764956 6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:08.773734 6972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:08.773806 6974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:08.773748 6971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:08.774037 6966 server_base.cc:1034] running on GCE node
I20260619 02:25:08.774621 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:08.775733 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:08.776933 6966 hybrid_clock.cc:648] HybridClock initialized: now 1781835908776917 us; error 35 us; skew 500 ppm
Jun 19 02:25:08 dist-test-slave-xkrt krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:08.779843 6966 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 02:25:08.780807 6966 webserver.cc:460] Webserver started at http://127.0.98.61:37031/ using document root <none> and password file <none>
I20260619 02:25:08.781366 6966 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:08.781461 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:08.781711 6966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:08.783715 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/data/instance:
uuid: "53fc572233564e31945ad52c925ba385"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "121ed6aed6cc14596ba70729ac6407e4"
server_key_iv: "272dbf9e05081d075303e9f480a4f862"
server_key_version: "encryptionkey@0"
I20260619 02:25:08.784261 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/wal/instance:
uuid: "53fc572233564e31945ad52c925ba385"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "121ed6aed6cc14596ba70729ac6407e4"
server_key_iv: "272dbf9e05081d075303e9f480a4f862"
server_key_version: "encryptionkey@0"
I20260619 02:25:08.787884 6966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:25:08.790263 6981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:08.791458 6966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:08.791599 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/wal
uuid: "53fc572233564e31945ad52c925ba385"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "121ed6aed6cc14596ba70729ac6407e4"
server_key_iv: "272dbf9e05081d075303e9f480a4f862"
server_key_version: "encryptionkey@0"
I20260619 02:25:08.791711 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:08.836282 6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:08.839711 6966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:08.839929 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:08.846486 6966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36913
I20260619 02:25:08.846823 7033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36913 every 8 connection(s)
I20260619 02:25:08.847462 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/data/info.pb
I20260619 02:25:08.850107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 6966
I20260619 02:25:08.850428 7034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:08.854903 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:25:08 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 02:25:08 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 02:25:08.865583 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.851674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.0.98.61:36913)
0619 02:25:08.852104 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:08.852119 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:08.852154 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:25:08.853112 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:08.853132 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:08.853461 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:08.853874 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:08.854634 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.855717 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.856630 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.857344 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.857492 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.860380 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:08.860385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:08.860393 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:08.860418 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:08.862285 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.862832 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.862837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.862842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.862949 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.863283 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.863286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.863288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.863658 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:08.863860 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:08.864183 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 02:25:08.864475 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":137,"threads_started":1}
I20260619 02:25:08.866437 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.856608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55010 (local address 127.0.98.62:40883)
0619 02:25:08.856743 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:08.856747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:08.857030 (+ 283us) server_negotiation.cc:407] Connection header received
0619 02:25:08.857150 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:08.857153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:08.857212 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:08.857285 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:08.858162 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.859038 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.860043 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.860389 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.860467 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.862998 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:08.863002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:08.863007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:08.863034 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:08.864646 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.865123 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.865126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.865128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.865199 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.865443 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.865446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.865447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.865714 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:08.865803 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:08.866170 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 02:25:08.866281 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:25:08.866526 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.856508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.0.1:55010)
0619 02:25:08.856899 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:08.856946 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:08.857056 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:08.857357 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:08.857365 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:08.857859 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:08.858032 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.858041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.859178 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.859181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.859904 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.859914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.860487 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.860493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.860633 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.861136 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:08.861153 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:08.862868 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:08.864741 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.864747 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.864753 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.865036 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.865266 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.865269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.865271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.865358 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.865858 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:08.865863 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:08.865975 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 02:25:08.866117 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 02:25:08.866285 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":138,"threads_started":1}
I20260619 02:25:08.867911 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:25:08 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 02:25:08.875202 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.868407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36913 (local address 127.0.0.1:52638)
0619 02:25:08.868537 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:08.868551 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:08.868625 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:08.868804 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:08.868806 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:08.868931 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:08.869015 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.869019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.869736 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.869738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.870286 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.870295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.870758 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.870761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.870867 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.871243 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:08.871254 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:08.872361 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:08.874044 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.874047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.874048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.874216 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.874432 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.874434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.874436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.874503 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.874805 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:08.874807 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:08.874878 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 02:25:08.874957 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:25:08.875046 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":21}
I20260619 02:25:08.875206 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.868493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.0.98.61:36913)
0619 02:25:08.868597 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:08.868600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:08.868611 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:25:08.868683 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:08.868685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:08.868716 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:08.868775 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:08.869102 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.869643 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.870379 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.870640 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.870744 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.872483 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:08.872486 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:08.872488 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:08.872507 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:08.873955 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.874293 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.874295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.874296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.874359 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.874574 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.874576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.874577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.874717 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:08.874790 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:08.875008 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 02:25:08.875088 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":39}
I20260619 02:25:08.878582 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: Bootstrap starting.
I20260619 02:25:08.881285 7034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:08.882107 7034 log.cc:824] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:08.883865 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: No bootstrap required, opened a new log
I20260619 02:25:08.886830 7034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } }
I20260619 02:25:08.887027 7034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:08.887063 7034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53fc572233564e31945ad52c925ba385, State: Initialized, Role: FOLLOWER
I20260619 02:25:08.887516 7034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } }
I20260619 02:25:08.887955 7040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } } }
I20260619 02:25:08.888103 7040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:25:08.888620 7034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:25:08.893247 7051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:25:08.893397 7051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:25:08.894940 6966 master_runner.cc:428] Detected that this master 53fc572233564e31945ad52c925ba385 is joining an existing cluster
I20260619 02:25:08.895002 6966 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:36913
I20260619 02:25:08.906642 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.897996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55018 (local address 127.0.98.62:40883)
0619 02:25:08.898106 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:08.898111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:08.898278 (+ 167us) server_negotiation.cc:407] Connection header received
0619 02:25:08.898422 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:08.898425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:08.898479 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:08.898554 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:08.899141 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.899975 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.901244 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.901609 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.901680 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.903094 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:08.903098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:08.903101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:08.903125 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:08.904874 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.905343 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.905346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.905348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.905422 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.905748 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.905751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.905753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.905954 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:08.906021 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:08.906346 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 02:25:08.906478 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:25:08.906697 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.897959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.0.98.61:36913)
0619 02:25:08.898068 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:08.898073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:08.898581 (+ 508us) server_negotiation.cc:407] Connection header received
0619 02:25:08.898707 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:08.898710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:08.898748 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:08.898792 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:08.899374 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.900115 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.901285 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.901657 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.901732 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.903186 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:08.903190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:08.903193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:08.903214 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:08.904992 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.905482 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.905485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.905487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.905584 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.905877 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.905882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.905885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.906109 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:08.906171 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:08.906453 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 02:25:08.906555 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:25:08.907094 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.897653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36913 (local address 127.0.0.1:52654)
0619 02:25:08.898517 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:08.898528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:08.898623 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:08.898907 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:08.898910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:08.899122 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:08.899247 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.899254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.900236 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.900239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.901165 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.901176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.901867 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.901872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.901976 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.902599 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:08.902614 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:08.903057 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:08.905105 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.905108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.905110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.905380 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.905689 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.905693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.905696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.905779 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.906224 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:08.906226 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:08.906308 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 02:25:08.906814 (+ 506us) client_negotiation.cc:240] Negotiation successful
0619 02:25:08.906934 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":34,"thread_start_us":89,"threads_started":1}
I20260619 02:25:08.907691 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.897590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.0.1:55018)
0619 02:25:08.898185 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:08.898199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:08.898315 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:08.898671 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:08.898674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:08.898882 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:08.899006 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.899015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.900109 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.900112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.901121 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.901130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.901836 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.901839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.901928 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.902440 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:08.902460 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:08.902946 (+ 486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:08.904991 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.904993 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.904996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.905242 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.905578 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.905581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.905583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.905658 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.906088 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:08.906090 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:08.906184 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 02:25:08.907334 (+ 1150us) client_negotiation.cc:240] Negotiation successful
0619 02:25:08.907511 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
Jun 19 02:25:08 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835907, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 02:25:08.921386 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.911864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.0.98.61:36913)
0619 02:25:08.911978 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:08.911982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:08.912308 (+ 326us) server_negotiation.cc:407] Connection header received
0619 02:25:08.912476 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:08.912480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:08.912533 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:08.912618 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:08.913163 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.913935 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.914880 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.915231 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.915344 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.917719 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:08.917722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:08.917725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:08.917769 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:08.919335 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.919918 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.919921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.919923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.920006 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.920379 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.920382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.920384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.920623 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:08.920720 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:08.921120 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 02:25:08.921227 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:25:08.921504 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.911820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36913 (local address 127.0.0.1:52656)
0619 02:25:08.912175 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:08.912222 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:08.912351 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:08.912653 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:08.912657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:08.912961 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:08.913058 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.913063 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.914033 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.914036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.914779 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.914788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.915338 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.915340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.915422 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.915942 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:08.915953 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:08.917517 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:08.919453 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.919493 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.919502 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.919814 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.920107 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.920113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.920118 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.920217 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.920740 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:08.920748 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:08.920894 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 02:25:08.921059 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 02:25:08.921286 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":126,"threads_started":1}
I20260619 02:25:08.923048 6502 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:36913: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "53fc572233564e31945ad52c925ba385"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 36913
}
attrs {
promote: true
}
}
dest_uuid: "e8e16d7c823b4c189a05e3c8bb416c35"
cas_config_opid_index: -1
I20260619 02:25:08.923734 6502 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: true } }
I20260619 02:25:08.925472 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 53fc572233564e31945ad52c925ba385 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: true } } }
I20260619 02:25:08.926141 7008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:08.927148 7062 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:36913
I20260619 02:25:08.927388 7008 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 FOLLOWER]: Refusing update from remote peer e8e16d7c823b4c189a05e3c8bb416c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 02:25:08.927420 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: true } } }
I20260619 02:25:08.927511 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:08.927994 6966 master.cc:537] Master@127.0.98.61:36913 shutting down...
I20260619 02:25:08.928051 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:08.932561 6966 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:08.932832 6966 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 02:25:08.932860 6487 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 02:25:08.932935 6966 tablet_replica.cc:330] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: stopping tablet replica
W20260619 02:25:08.933784 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 -> Peer 53fc572233564e31945ad52c925ba385 (127.0.98.61:36913): Couldn't send request to peer 53fc572233564e31945ad52c925ba385. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 02:25:08.937708 6966 master.cc:552] Master@127.0.98.61:36913 shutdown complete.
I20260619 02:25:08.937773 6966 master_runner.cc:305] Clearing existing system tablet
I20260619 02:25:08.938014 6966 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:08.938057 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260619 02:25:08.939453 6966 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 02:25:08.940491 7067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:08.940699 6966 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:08.940850 6966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:25:08.940903 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/wal
uuid: "53fc572233564e31945ad52c925ba385"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "121ed6aed6cc14596ba70729ac6407e4"
server_key_iv: "272dbf9e05081d075303e9f480a4f862"
server_key_version: "encryptionkey@0"
I20260619 02:25:08.942639 6966 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:08.944689 6966 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 02:25:08.944813 6966 log.cc:1196] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 02:25:08.945093 6966 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: Deleting consensus metadata
I20260619 02:25:08.945257 6966 master_runner.cc:315] Copying system tablet from 127.0.98.62:40883
I20260619 02:25:08.946410 6966 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40883
I20260619 02:25:08.955221 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.947065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55028 (local address 127.0.98.62:40883)
0619 02:25:08.947219 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:08.947225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:08.947349 (+ 124us) server_negotiation.cc:407] Connection header received
0619 02:25:08.947462 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:08.947466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:08.947529 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:08.947637 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:08.948129 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.949030 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.950338 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.950641 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.950727 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.951973 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:08.951977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:08.951978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:08.952000 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:08.953505 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.954010 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.954012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.954014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.954085 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:08.954429 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:08.954433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:08.954434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:08.954626 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:08.954716 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:08.955013 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 02:25:08.955108 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:25:08.955708 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:08.946895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.0.1:55028)
0619 02:25:08.947234 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:08.947249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:08.947371 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:08.947661 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:08.947664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:08.947875 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:08.947991 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.947998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.949146 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.949149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.950202 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:08.950212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:08.950753 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:08.950757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:08.950851 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:08.951404 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:08.951418 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:08.951817 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:08.953656 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.953660 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.953662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.953906 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.954167 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:08.954170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:08.954172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:08.954261 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:08.954724 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:08.954727 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:08.954825 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 02:25:08.954967 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 02:25:08.955106 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":89,"threads_started":1}
I20260619 02:25:08.956238 6522 tablet_copy_service.cc:140] P e8e16d7c823b4c189a05e3c8bb416c35: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 53fc572233564e31945ad52c925ba385 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:55028)
I20260619 02:25:08.956425 6522 tablet_copy_service.cc:161] P e8e16d7c823b4c189a05e3c8bb416c35: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 53fc572233564e31945ad52c925ba385 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:55028: session id = 53fc572233564e31945ad52c925ba385-00000000000000000000000000000000
I20260619 02:25:08.957993 6522 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35: Tablet Copy: opened 0 blocks and 1 log segments
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 02:25:08.959045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45005
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40883,127.0.98.61:36913,127.0.98.60:45005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:25:08.959663 6966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 02:25:08.961884 6966 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: tablet copy: Starting download of 0 data blocks...
I20260619 02:25:08.962059 6966 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: tablet copy: Starting download of 1 WAL segments...
I20260619 02:25:08.963995 6966 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:25:08.965907 6522 tablet_copy_service.cc:342] P e8e16d7c823b4c189a05e3c8bb416c35: Request end of tablet copy session 53fc572233564e31945ad52c925ba385-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:55028
I20260619 02:25:08.966023 6522 tablet_copy_service.cc:433] P e8e16d7c823b4c189a05e3c8bb416c35: ending tablet copy session 53fc572233564e31945ad52c925ba385-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 53fc572233564e31945ad52c925ba385
I20260619 02:25:08.974098 6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:08.975855 7075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:08.976197 7078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:08.976388 6966 server_base.cc:1034] running on GCE node
W20260619 02:25:08.977718 7076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:08.977977 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:08.978282 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:08.979478 6966 hybrid_clock.cc:648] HybridClock initialized: now 1781835908979467 us; error 30 us; skew 500 ppm
Jun 19 02:25:08 dist-test-slave-xkrt krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:08.981117 6966 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 02:25:08.981570 6966 webserver.cc:460] Webserver started at http://127.0.98.61:45303/ using document root <none> and password file <none>
I20260619 02:25:08.981771 6966 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:08.981818 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:08.985931 6966 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260619 02:25:08.986933 7085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:08.987159 6966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:25:08.987232 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/wal
uuid: "53fc572233564e31945ad52c925ba385"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "121ed6aed6cc14596ba70729ac6407e4"
server_key_iv: "272dbf9e05081d075303e9f480a4f862"
server_key_version: "encryptionkey@0"
I20260619 02:25:08.987324 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:09.001448 6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:09.003566 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:09.008260 6966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36913
I20260619 02:25:09.008512 7137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36913 every 8 connection(s)
I20260619 02:25:09.008888 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-1/data/info.pb
I20260619 02:25:09.010654 7138 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:25:09.011641 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: Bootstrap starting.
I20260619 02:25:09.021612 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:25:09.021926 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: Bootstrap complete.
I20260619 02:25:09.022516 7138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: true } }
I20260619 02:25:09.022653 7138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 53fc572233564e31945ad52c925ba385, State: Initialized, Role: LEARNER
I20260619 02:25:09.022797 7138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: true } }
I20260619 02:25:09.023126 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: true } } }
I20260619 02:25:09.023236 7138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:09.023370 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:25:09.026489 7152 catalog_manager.cc:1259] Loaded cluster ID: 9d73f6d34c9c4fa8aeb90a06b2f1baa6
I20260619 02:25:09.026548 7152 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: loading cluster ID for follower catalog manager: success
I20260619 02:25:09.028611 7152 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: acquiring CA information for follower catalog manager: success
I20260619 02:25:09.029987 7152 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 02:25:09.084738 7074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:09.085038 7074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:09.085108 7074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:09.089320 7074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:09.089402 7074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:09.089438 7074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:09.089494 7074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:09.089563 7074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:25:09.095160 7074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40883,127.0.98.61:36913,127.0.98.60:45005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:09.096225 7074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:09.097229 7074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:09.105578 7158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:09.105510 7157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:09.105482 7160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:09.105748 7074 server_base.cc:1034] running on GCE node
I20260619 02:25:09.106328 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:09.107383 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:09.108536 7074 hybrid_clock.cc:648] HybridClock initialized: now 1781835909108518 us; error 33 us; skew 500 ppm
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:09.111328 7074 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 02:25:09.112293 7074 webserver.cc:460] Webserver started at http://127.0.98.60:44643/ using document root <none> and password file <none>
I20260619 02:25:09.112844 7074 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:09.112931 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:09.113164 7074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:09.115412 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/data/instance:
uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555"
format_stamp: "Formatted at 2026-06-19 02:25:09 on dist-test-slave-xkrt"
server_key: "cc7b30019202729bca91fcf93f4baf52"
server_key_iv: "fb2b82aafcd800abe078cc24fde21607"
server_key_version: "encryptionkey@0"
I20260619 02:25:09.116007 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/wal/instance:
uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555"
format_stamp: "Formatted at 2026-06-19 02:25:09 on dist-test-slave-xkrt"
server_key: "cc7b30019202729bca91fcf93f4baf52"
server_key_iv: "fb2b82aafcd800abe078cc24fde21607"
server_key_version: "encryptionkey@0"
I20260619 02:25:09.119572 7074 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260619 02:25:09.121937 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:09.123126 7074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:09.123277 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/wal
uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555"
format_stamp: "Formatted at 2026-06-19 02:25:09 on dist-test-slave-xkrt"
server_key: "cc7b30019202729bca91fcf93f4baf52"
server_key_iv: "fb2b82aafcd800abe078cc24fde21607"
server_key_version: "encryptionkey@0"
I20260619 02:25:09.123394 7074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:09.159432 7074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:09.162815 7074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:09.163029 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:09.171324 7074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45005
I20260619 02:25:09.171388 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45005 every 8 connection(s)
I20260619 02:25:09.172375 7074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/data/info.pb
I20260619 02:25:09.175194 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:09.176007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 7074
I20260619 02:25:09.181123 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 02:25:09.192090 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.177447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.0.98.60:45005)
0619 02:25:09.177928 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.177946 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.177986 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:25:09.178810 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.178841 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.179266 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.179630 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.180318 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.181851 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.182781 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.183505 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.184095 (+ 590us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.186947 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.186953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.186965 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.186994 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.189069 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.189563 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.189569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.189575 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.189702 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.190037 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.190042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.190043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.190505 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.190701 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.191010 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.191306 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":143,"threads_started":1}
I20260619 02:25:09.194098 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.183172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.0.98.62:40883)
0619 02:25:09.183307 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.183312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.183580 (+ 268us) server_negotiation.cc:407] Connection header received
0619 02:25:09.183720 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.183722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.183764 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.183832 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.184602 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.185502 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.186898 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.187243 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.187342 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.189929 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.189932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.189935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.189957 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.191412 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.192021 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.192024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.192026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.192112 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.193166 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.193170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.193171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.193355 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.193464 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.193832 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.193952 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:25:09.194379 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.182977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.0.1:55030)
0619 02:25:09.183440 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.183490 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.183619 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.183916 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.183923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.184363 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.184476 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.184481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.185672 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.185676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.186735 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.186743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.187410 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.187415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.187569 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.188134 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.188152 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.189815 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.191646 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.191653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.191658 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.191910 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.192254 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.192259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.192972 (+ 713us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.193051 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.193460 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.193466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.193637 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.193782 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.194223 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":143,"threads_started":1}
I20260619 02:25:09.195423 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:25:09.199991 6693 heartbeater.cc:503] Master 127.0.98.62:40883 was elected leader, sending a full tablet report...
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 02:25:09.204145 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.195988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36913 (local address 127.0.0.1:52662)
0619 02:25:09.196338 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.196350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.196411 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.196628 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.196630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.196756 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.196840 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.196845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.197890 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.197893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.198437 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.198445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.198907 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.198910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.199150 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.199662 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.199674 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.201044 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.202927 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.202929 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.202931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.203115 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.203365 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.203367 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.203368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.203437 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.203778 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.203780 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.203852 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.203929 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.204032 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293}
I20260619 02:25:09.204206 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.196140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.0.98.61:36913)
0619 02:25:09.196434 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.196437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.196451 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:25:09.196487 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.196489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.196525 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.196602 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.196942 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.197775 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.198549 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.198794 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.198879 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.201214 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.201218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.201221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.201248 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.202818 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.203199 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.203201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.203202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.203280 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.203529 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.203531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.203532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.203702 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.203774 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.203993 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.204087 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":82,"threads_started":1}
I20260619 02:25:09.205008 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 02:25:09.212195 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.205459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45005 (local address 127.0.0.1:48536)
0619 02:25:09.205603 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.205616 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.205685 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.205872 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.205874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.205995 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.206059 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.206063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.207088 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.207090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.207582 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.207588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.207975 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.207977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.208048 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.208494 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.208506 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.209660 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.211084 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.211086 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.211088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.211263 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.211486 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.211488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.211489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.211556 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.211854 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.211856 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.211921 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.211997 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.212085 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":10}
I20260619 02:25:09.212297 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.205567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.0.98.60:45005)
0619 02:25:09.205656 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.205659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.205669 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:25:09.205748 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.205750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.205781 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.205843 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.206147 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.206991 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.207673 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.207883 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.207975 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.209777 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.209780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.209782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.209800 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.210991 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.211343 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.211345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.211346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.211409 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.211627 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.211630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.211630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.211777 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.211850 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.212059 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.212141 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 02:25:09.215188 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: Bootstrap starting.
I20260619 02:25:09.217443 7220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:09.218595 7220 log.cc:824] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:09.220429 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: No bootstrap required, opened a new log
I20260619 02:25:09.223397 7220 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } }
I20260619 02:25:09.223695 7220 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:09.223783 7220 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9bc13ecd1524b5e94c4e1fe5e75a555, State: Initialized, Role: FOLLOWER
I20260619 02:25:09.224278 7220 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } }
I20260619 02:25:09.224797 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } } }
I20260619 02:25:09.224910 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:25:09.225581 7220 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:25:09.230382 7237 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:25:09.230464 7237 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:25:09.233330 7074 master_runner.cc:428] Detected that this master f9bc13ecd1524b5e94c4e1fe5e75a555 is joining an existing cluster
I20260619 02:25:09.233388 7074 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:45005
I20260619 02:25:09.244303 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.236578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.0.98.60:45005)
0619 02:25:09.236670 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.236673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.236890 (+ 217us) server_negotiation.cc:407] Connection header received
0619 02:25:09.237036 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.237040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.237079 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.237126 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.237687 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.238376 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.239575 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.239896 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.239963 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.241266 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.241270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.241272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.241293 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.242728 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.243135 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.243141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.243143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.243218 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.243495 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.243501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.243503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.243674 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.243777 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.244006 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.244150 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":38}
I20260619 02:25:09.244318 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.236284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45005 (local address 127.0.0.1:48544)
0619 02:25:09.236817 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.236832 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.236942 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.237273 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.237276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.237469 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.237587 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.237594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.238489 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.238493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.239458 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.239468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.240114 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.240117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.240210 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.240764 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.240792 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.241122 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.242842 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.242845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.242847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.243043 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.243319 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.243323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.243325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.243404 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.243768 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.243771 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.243851 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.243963 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.244100 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":26,"thread_start_us":86,"threads_started":1}
I20260619 02:25:09.245225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 6564
I20260619 02:25:09.251446 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36963
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40883
--webserver_port=39305
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40883,127.0.98.61:36913
--tserver_master_addrs=127.0.98.62:40883,127.0.98.61:36913,127.0.98.60:45005 with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:25:09.263319 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.235828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.0.98.61:36913)
0619 02:25:09.235940 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.235944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.236316 (+ 372us) server_negotiation.cc:407] Connection header received
0619 02:25:09.236493 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.236498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.236547 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.236599 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.237457 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.238308 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.239755 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.240118 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.240194 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.259116 (+ 18922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.259121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.259124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.259156 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.261278 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.261816 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.261819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.261821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.261902 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.262195 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.262198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.262199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.262543 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.262625 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.262982 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.263132 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:25:09.263341 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.235769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.0.98.62:40883)
0619 02:25:09.235886 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.235891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.236194 (+ 303us) server_negotiation.cc:407] Connection header received
0619 02:25:09.236393 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.236397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.236448 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.236504 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.237450 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.238321 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.239660 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.239997 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.240070 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.259150 (+ 19080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.259156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.259162 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.259198 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.261060 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.261738 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.261742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.261746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.261842 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.262181 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.262184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.262186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.262428 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.262582 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.263003 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.263127 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:25:09.263856 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.235677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36913 (local address 127.0.0.1:52674)
0619 02:25:09.236212 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.236226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.236382 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.236737 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.236740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.237014 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.237315 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.237323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.238451 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.238454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.239603 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.239613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.240362 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.240365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.240466 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.258301 (+ 17835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.258317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.258925 (+ 608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.261406 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.261409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.261411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.261694 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.262008 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.262011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.262013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.262099 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.262703 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.262706 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.262803 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.263508 (+ 705us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.263668 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":450,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260619 02:25:09.265522 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.235677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.0.1:55034)
0619 02:25:09.236109 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.236124 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.236271 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.236658 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.236661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.237010 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.237320 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.237330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.238452 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.238455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.239531 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.239541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.240204 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.240207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.240292 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.258190 (+ 17898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.258214 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.258920 (+ 706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.261256 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.261260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.261264 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.261605 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.261981 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.261984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.261986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.262068 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.262709 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.262712 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.262817 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.265163 (+ 2346us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.265324 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":199,"spinlock_wait_cycles":5376,"thread_start_us":109,"threads_started":1}
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835907, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 02:25:09.278642 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.269674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45005 (local address 127.0.0.1:48554)
0619 02:25:09.269834 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.269848 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.269931 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.270211 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.270214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.270431 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.270554 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.270560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.271595 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.271598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.272499 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.272509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.273090 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.273093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.273189 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.273691 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.273706 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.275217 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.277051 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.277054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.277056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.277305 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.277626 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.277630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.277632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.277724 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.278128 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.278131 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.278225 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.278344 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.278481 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 02:25:09.278733 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.269711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.0.98.60:45005)
0619 02:25:09.269815 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.269819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.269892 (+ 73us) server_negotiation.cc:407] Connection header received
0619 02:25:09.270049 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.270052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.270100 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.270179 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.270657 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.271471 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.272617 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.272976 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.273076 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.275378 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.275382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.275385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.275407 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.276935 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.277404 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.277408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.277409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.277487 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.277819 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.277822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.277823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.278030 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.278124 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.278458 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.278565 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:25:09.279990 6502 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:45005: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 45005
}
attrs {
promote: true
}
}
dest_uuid: "e8e16d7c823b4c189a05e3c8bb416c35"
cas_config_opid_index: 5
I20260619 02:25:09.280561 6502 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: true } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } }
I20260619 02:25:09.281867 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f9bc13ecd1524b5e94c4e1fe5e75a555 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: true } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } } }
I20260619 02:25:09.283318 7061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:45005
I20260619 02:25:09.283713 7074 master.cc:537] Master@127.0.98.60:45005 shutting down...
I20260619 02:25:09.285437 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: true } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } } }
W20260619 02:25:09.285566 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 -> Peer f9bc13ecd1524b5e94c4e1fe5e75a555 (127.0.98.60:45005): Couldn't send request to peer f9bc13ecd1524b5e94c4e1fe5e75a555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 02:25:09.287731 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:09.288246 7074 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:09.288518 7074 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:09.288612 7074 tablet_replica.cc:330] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: stopping tablet replica
I20260619 02:25:09.291002 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.283940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.0.98.61:36913)
0619 02:25:09.284163 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.284167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.285190 (+ 1023us) server_negotiation.cc:407] Connection header received
0619 02:25:09.286047 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.286050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.286111 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.286171 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:09.286536 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.287336 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.289355 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.290010 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.290089 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.290230 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.290776 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.290857 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260619 02:25:09.291828 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.283719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36913 (local address 127.0.0.1:52686)
0619 02:25:09.285036 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.285051 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.285915 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.286330 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.286332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.286347 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:09.286425 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.286430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.287505 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.287508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.289203 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.289214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.290488 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.290493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.290606 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.290650 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.291360 (+ 710us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.291704 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1256}
I20260619 02:25:09.292616 7112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 LEARNER]: Refusing update from remote peer e8e16d7c823b4c189a05e3c8bb416c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 02:25:09.293298 7247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 02:25:09.293680 7074 master.cc:552] Master@127.0.98.60:45005 shutdown complete.
I20260619 02:25:09.293726 7074 master_runner.cc:305] Clearing existing system tablet
I20260619 02:25:09.293963 7074 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:09.294006 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:09.295167 7074 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 02:25:09.295218 7112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f9bc13ecd1524b5e94c4e1fe5e75a555 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: true } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } } }
I20260619 02:25:09.295923 7250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:09.296074 7074 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:09.296166 7074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:25:09.296195 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/wal
uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555"
format_stamp: "Formatted at 2026-06-19 02:25:09 on dist-test-slave-xkrt"
server_key: "cc7b30019202729bca91fcf93f4baf52"
server_key_iv: "fb2b82aafcd800abe078cc24fde21607"
server_key_version: "encryptionkey@0"
I20260619 02:25:09.297513 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8e16d7c823b4c189a05e3c8bb416c35. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: true } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } } }
I20260619 02:25:09.297650 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:25:09.297739 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: true } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } } }
I20260619 02:25:09.297804 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:25:09.298151 7074 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:09.300377 7074 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 02:25:09.300462 7074 log.cc:1196] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 02:25:09.300705 7074 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: Deleting consensus metadata
I20260619 02:25:09.300865 7074 master_runner.cc:315] Copying system tablet from 127.0.98.62:40883
I20260619 02:25:09.302421 7074 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40883
I20260619 02:25:09.302908 7247 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35: attempting to promote NON_VOTER 53fc572233564e31945ad52c925ba385 to VOTER
I20260619 02:25:09.303361 7247 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } }
I20260619 02:25:09.305234 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.304894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45005 (local address 127.0.0.1:48556)
0619 02:25:09.305092 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.305158 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260619 02:25:09.305712 7112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 LEARNER]: Refusing update from remote peer e8e16d7c823b4c189a05e3c8bb416c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 02:25:09.306420 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 02:25:09.307126 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.306819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45005 (local address 127.0.0.1:48568)
0619 02:25:09.307015 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.307062 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260619 02:25:09.308202 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 -> Peer f9bc13ecd1524b5e94c4e1fe5e75a555 (127.0.98.60:45005): Couldn't send request to peer f9bc13ecd1524b5e94c4e1fe5e75a555. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 02:25:09.308758 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 53fc572233564e31945ad52c925ba385 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } } }
I20260619 02:25:09.310140 7112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 53fc572233564e31945ad52c925ba385 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } } }
I20260619 02:25:09.310626 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } } }
I20260619 02:25:09.310737 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:09.310845 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } } }
I20260619 02:25:09.310917 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:09.313220 7251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } } }
I20260619 02:25:09.313361 7251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:25:09.314946 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.304335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.0.98.62:40883)
0619 02:25:09.304447 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.304451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.305831 (+ 1380us) server_negotiation.cc:407] Connection header received
0619 02:25:09.306007 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.306010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.306052 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.306099 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.306683 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.307485 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.308589 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.308915 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.308981 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.310588 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.310591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.310594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.310618 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.313034 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.313628 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.313632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.313633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.313720 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.314048 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.314051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.314053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.314284 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.314377 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.314698 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.314811 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:25:09.315461 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.304273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.0.1:55038)
0619 02:25:09.305751 (+ 1478us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.305764 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.305904 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.306229 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.306231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.306459 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.306558 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.306565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.307618 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.307621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.308467 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.308477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.309125 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.309128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.309208 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.309943 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.309960 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.310440 (+ 480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.313142 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.313146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.313148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.313499 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.313815 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.313818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.313820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.313908 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.314423 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.314426 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.314524 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.315108 (+ 584us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.315269 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1401,"thread_start_us":70,"threads_started":1}
I20260619 02:25:09.315832 6522 tablet_copy_service.cc:140] P e8e16d7c823b4c189a05e3c8bb416c35: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f9bc13ecd1524b5e94c4e1fe5e75a555 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:55038)
I20260619 02:25:09.315922 6522 tablet_copy_service.cc:161] P e8e16d7c823b4c189a05e3c8bb416c35: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f9bc13ecd1524b5e94c4e1fe5e75a555 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:55038: session id = f9bc13ecd1524b5e94c4e1fe5e75a555-00000000000000000000000000000000
I20260619 02:25:09.316900 6522 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 02:25:09.318156 7074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 02:25:09.320412 7074 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: tablet copy: Starting download of 0 data blocks...
I20260619 02:25:09.320590 7074 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: tablet copy: Starting download of 1 WAL segments...
I20260619 02:25:09.321936 7074 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:25:09.323597 6522 tablet_copy_service.cc:342] P e8e16d7c823b4c189a05e3c8bb416c35: Request end of tablet copy session f9bc13ecd1524b5e94c4e1fe5e75a555-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:55038
I20260619 02:25:09.323668 6522 tablet_copy_service.cc:433] P e8e16d7c823b4c189a05e3c8bb416c35: ending tablet copy session f9bc13ecd1524b5e94c4e1fe5e75a555-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f9bc13ecd1524b5e94c4e1fe5e75a555
I20260619 02:25:09.330590 7074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:09.332326 7259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:09.332392 7258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:09.334667 7261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:09.334908 7074 server_base.cc:1034] running on GCE node
I20260619 02:25:09.335052 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:09.335338 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:09.336472 7074 hybrid_clock.cc:648] HybridClock initialized: now 1781835909336462 us; error 28 us; skew 500 ppm
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:09.338091 7074 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 02:25:09.338527 7074 webserver.cc:460] Webserver started at http://127.0.98.60:33889/ using document root <none> and password file <none>
I20260619 02:25:09.338716 7074 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:09.338757 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:09.340056 7074 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:25:09.341318 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:09.341692 7074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:25:09.341769 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/wal
uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555"
format_stamp: "Formatted at 2026-06-19 02:25:09 on dist-test-slave-xkrt"
server_key: "cc7b30019202729bca91fcf93f4baf52"
server_key_iv: "fb2b82aafcd800abe078cc24fde21607"
server_key_version: "encryptionkey@0"
I20260619 02:25:09.341856 7074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:09.350965 7074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:09.352993 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:09.357508 7074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45005
I20260619 02:25:09.357532 7320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45005 every 8 connection(s)
I20260619 02:25:09.358094 7074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/master-2/data/info.pb
I20260619 02:25:09.359815 7321 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:25:09.360762 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: Bootstrap starting.
I20260619 02:25:09.372139 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:25:09.372485 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: Bootstrap complete.
I20260619 02:25:09.373051 7321 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } }
I20260619 02:25:09.373198 7321 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f9bc13ecd1524b5e94c4e1fe5e75a555, State: Initialized, Role: LEARNER
I20260619 02:25:09.373297 7321 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } }
I20260619 02:25:09.373663 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } } }
I20260619 02:25:09.373795 7321 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:09.373870 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:25:09.377283 7335 catalog_manager.cc:1259] Loaded cluster ID: 9d73f6d34c9c4fa8aeb90a06b2f1baa6
I20260619 02:25:09.377417 7335 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: loading cluster ID for follower catalog manager: success
I20260619 02:25:09.379669 7335 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: acquiring CA information for follower catalog manager: success
I20260619 02:25:09.381194 7335 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 02:25:09.393082 7246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:09.393428 7246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:09.393512 7246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:09.397785 7246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:09.397842 7246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:09.397933 7246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:25:09.403215 7246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39305
--tserver_master_addrs=127.0.98.62:40883,127.0.98.61:36913,127.0.98.60:45005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:09.404284 7246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:09.405315 7246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:25:09.407330 6827 heartbeater.cc:503] Master 127.0.98.62:40883 was elected leader, sending a full tablet report...
W20260619 02:25:09.415200 7340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:09.415321 7341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:09.415549 7246 server_base.cc:1034] running on GCE node
W20260619 02:25:09.415198 7343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:09.415951 7246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:09.416545 7246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:09.417645 7246 hybrid_clock.cc:648] HybridClock initialized: now 1781835909417624 us; error 37 us; skew 500 ppm
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:09.420390 7246 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 02:25:09.421274 7246 webserver.cc:460] Webserver started at http://127.0.98.1:39305/ using document root <none> and password file <none>
I20260619 02:25:09.421883 7246 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:09.421958 7246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:09.426126 7246 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 02:25:09.428212 7350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:09.429422 7246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:09.429518 7246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "990e7d22b7cd4a0baefea77792def123"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "79d0ca1cae75a2f4a922b6cbe4f5cba0"
server_key_iv: "fd8e5ace6afc20d3369e34f659ad635e"
server_key_version: "encryptionkey@0"
I20260619 02:25:09.430091 7246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:09.467435 7246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:09.470745 7246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:09.470961 7246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:09.471635 7246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:09.472522 7246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:09.472568 7246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:09.472607 7246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:09.472623 7246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:09.482496 7246 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36963
I20260619 02:25:09.482576 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36963 every 8 connection(s)
I20260619 02:25:09.483560 7246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:25:09.491040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 7246
I20260619 02:25:09.491245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 6698
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 02:25:09.499372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36889
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40883
--webserver_port=41203
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40883,127.0.98.61:36913
--tserver_master_addrs=127.0.98.62:40883,127.0.98.61:36913,127.0.98.60:45005 with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:25:09.501775 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.487220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59741 (local address 127.0.98.61:36913)
0619 02:25:09.487382 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.487386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.487401 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:25:09.488400 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.488407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.488471 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.488542 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.489801 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.490757 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.492795 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.493222 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.493307 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.496947 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.496951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.496955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.496989 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.499195 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.499839 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.499845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.499850 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.499941 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.500360 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.500366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.500371 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.500627 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.500713 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.501431 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.501596 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 02:25:09.501878 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.485390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60855 (local address 127.0.98.62:40883)
0619 02:25:09.485521 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.485526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.486710 (+ 1184us) server_negotiation.cc:407] Connection header received
0619 02:25:09.487963 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.487968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.488035 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.488126 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.491151 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.492030 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.493708 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.494063 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.494172 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.497187 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.497193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.497198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.497238 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.499928 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.500506 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.500512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.500515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.500608 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.500940 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.500943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.500945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.501154 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.501252 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.501609 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.501724 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:25:09.503046 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.485416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58121 (local address 127.0.98.60:45005)
0619 02:25:09.485727 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.485751 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.486968 (+ 1217us) server_negotiation.cc:407] Connection header received
0619 02:25:09.488064 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.488070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.488135 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.488220 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.489866 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.490983 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.493422 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.493844 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.493943 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.498255 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.498261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.498265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.498298 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.500559 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.501079 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.501087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.501092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.501198 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.501612 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.501617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.501621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.502199 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.502358 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.502662 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.502815 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":104,"threads_started":1}
I20260619 02:25:09.503573 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.486905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36913 (local address 127.0.98.1:59741)
0619 02:25:09.487176 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.487189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.487714 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.488588 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.488590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.488962 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.489641 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.489654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.490919 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.490928 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.492650 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.492664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.493529 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.493558 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.493817 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.494512 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.494549 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.496698 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.499329 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.499335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.499350 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.499705 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.500054 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.500060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.500065 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.500250 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.500808 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.500813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.501154 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.501952 (+ 798us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.502399 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":550,"spinlock_wait_cycles":22144,"thread_start_us":95,"threads_started":1}
I20260619 02:25:09.503875 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.485914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45005 (local address 127.0.98.1:58121)
0619 02:25:09.486876 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.486891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.487720 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.488654 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.488656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.488970 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.489634 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.489652 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.491102 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.491106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.492653 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.492666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.494070 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.494074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.494514 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.495850 (+ 1336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.495874 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.497680 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.500689 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.500692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.500694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.500958 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.501341 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.501347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.501349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.501460 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.502347 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.502352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.502448 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.502544 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.502660 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":87,"spinlock_wait_cycles":2304,"thread_start_us":119,"threads_started":1}
I20260619 02:25:09.504150 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.485914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.1:60855)
0619 02:25:09.486498 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.486566 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.487714 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.488370 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.488381 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.488962 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.489646 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.489654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.492181 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.492185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.493567 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.493578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.494369 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.494374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.494497 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.495183 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.495204 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.497016 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.500081 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.500084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.500088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.500372 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.500720 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.500725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.500730 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.500813 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.501294 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.501296 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.501407 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.502038 (+ 631us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.504027 (+ 1989us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":1053,"spinlock_wait_cycles":2432,"thread_start_us":687,"threads_started":1}
I20260619 02:25:09.505141 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40883
I20260619 02:25:09.505152 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.60:45005
I20260619 02:25:09.505388 7465 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:09.505390 7464 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:09.505901 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36913
I20260619 02:25:09.505990 7466 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:09.506004 7465 heartbeater.cc:511] Master 127.0.98.62:40883 requested a full tablet report, sending...
I20260619 02:25:09.506004 7464 heartbeater.cc:511] Master 127.0.98.60:45005 requested a full tablet report, sending...
I20260619 02:25:09.506161 7466 heartbeater.cc:511] Master 127.0.98.61:36913 requested a full tablet report, sending...
I20260619 02:25:09.507248 6502 ts_manager.cc:194] Re-registered known tserver with Master: 990e7d22b7cd4a0baefea77792def123 (127.0.98.1:36963)
I20260619 02:25:09.507841 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60855
I20260619 02:25:09.507872 7102 ts_manager.cc:194] Registered new tserver with Master: 990e7d22b7cd4a0baefea77792def123 (127.0.98.1:36963)
I20260619 02:25:09.509075 7277 ts_manager.cc:194] Registered new tserver with Master: 990e7d22b7cd4a0baefea77792def123 (127.0.98.1:36963)
I20260619 02:25:09.617020 6961 heartbeater.cc:503] Master 127.0.98.62:40883 was elected leader, sending a full tablet report...
W20260619 02:25:09.627334 7472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:09.627596 7472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:09.627681 7472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:09.631955 7472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:09.632010 7472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:09.632115 7472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:25:09.637405 7472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41203
--tserver_master_addrs=127.0.98.62:40883,127.0.98.61:36913,127.0.98.60:45005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:09.638441 7472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:09.639454 7472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:09.649418 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:09.649406 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:09.649587 7472 server_base.cc:1034] running on GCE node
W20260619 02:25:09.649621 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:09.650090 7472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:09.650760 7472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:09.651942 7472 hybrid_clock.cc:648] HybridClock initialized: now 1781835909651920 us; error 38 us; skew 500 ppm
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:09.654783 7472 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 02:25:09.655661 7472 webserver.cc:460] Webserver started at http://127.0.98.2:41203/ using document root <none> and password file <none>
I20260619 02:25:09.656217 7472 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:09.656288 7472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:09.660171 7472 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:09.662338 7487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:09.663626 7472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:09.663735 7472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "12458f15b67c479ca476b132816f5ac5"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "72a13a32462f88f2c170d1c8bcd250d2"
server_key_iv: "37cde3b7c51502b5e83c5365f4268985"
server_key_version: "encryptionkey@0"
I20260619 02:25:09.664322 7472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:09.676797 7472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:09.679936 7472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:09.680125 7472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:09.680725 7472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:09.681653 7472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:09.681701 7472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:09.681773 7472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:09.681805 7472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:09.690840 7472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36889
I20260619 02:25:09.690884 7600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36889 every 8 connection(s)
I20260619 02:25:09.691932 7472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:25:09.698444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 7472
I20260619 02:25:09.698634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 6832
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 02:25:09.706187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41937
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40883
--webserver_port=35173
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40883,127.0.98.61:36913
--tserver_master_addrs=127.0.98.62:40883,127.0.98.61:36913,127.0.98.60:45005 with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:25:09.709048 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.693861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36035 (local address 127.0.98.61:36913)
0619 02:25:09.694015 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.694020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.695109 (+ 1089us) server_negotiation.cc:407] Connection header received
0619 02:25:09.696179 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.696182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.696246 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.696314 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.698202 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.698949 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.701013 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.701457 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.701601 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.704879 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.704886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.704892 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.704926 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.706651 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.707280 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.707285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.707286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.707386 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.707799 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.707804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.707807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.708051 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.708136 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.708746 (+ 610us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.708870 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:25:09.709050 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.695534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54857 (local address 127.0.98.60:45005)
0619 02:25:09.695659 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.695663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.696397 (+ 734us) server_negotiation.cc:407] Connection header received
0619 02:25:09.696476 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.696479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.696523 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.696600 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.698333 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.699161 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.700822 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.701120 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.701178 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.704570 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.704577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.704584 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.704621 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.706834 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.707424 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.707431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.707436 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.707534 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.707903 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.707919 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.707921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.708134 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.708196 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.708744 (+ 548us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.708881 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 02:25:09.710126 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.695393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45005 (local address 127.0.98.2:54857)
0619 02:25:09.696280 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.696290 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.696404 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.696624 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.696626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.697127 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.698202 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.698212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.699315 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.699318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.700686 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.700695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.701347 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.701349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.701485 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.702577 (+ 1092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.702590 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.704373 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.706984 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.706991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.706997 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.707301 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.707673 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.707677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.707682 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.707784 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.708259 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.708261 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.708494 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.708704 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.709054 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":73,"spinlock_wait_cycles":28800,"thread_start_us":79,"threads_started":1}
I20260619 02:25:09.710461 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.694277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36913 (local address 127.0.98.2:36035)
0619 02:25:09.694907 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.694974 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.695976 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.696549 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.696556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.697126 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.697951 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.697970 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.700060 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.700063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.700892 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.700900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.701608 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.701611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.701707 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.702425 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.702451 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.704674 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.706788 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.706794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.706810 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.707168 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.707491 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.707496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.707499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.707684 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.708222 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.708229 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.708510 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.709253 (+ 743us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.709415 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":142,"spinlock_wait_cycles":14848,"thread_start_us":709,"threads_started":1}
I20260619 02:25:09.711692 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.60:45005
I20260619 02:25:09.711784 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36913
I20260619 02:25:09.711956 7603 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:09.711954 7601 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:09.712651 7603 heartbeater.cc:511] Master 127.0.98.61:36913 requested a full tablet report, sending...
I20260619 02:25:09.712651 7601 heartbeater.cc:511] Master 127.0.98.60:45005 requested a full tablet report, sending...
I20260619 02:25:09.713971 7277 ts_manager.cc:194] Registered new tserver with Master: 12458f15b67c479ca476b132816f5ac5 (127.0.98.2:36889)
I20260619 02:25:09.714049 7102 ts_manager.cc:194] Registered new tserver with Master: 12458f15b67c479ca476b132816f5ac5 (127.0.98.2:36889)
I20260619 02:25:09.714812 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.694526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52433 (local address 127.0.98.62:40883)
0619 02:25:09.694673 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.694678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.695721 (+ 1043us) server_negotiation.cc:407] Connection header received
0619 02:25:09.696472 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.696475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.696521 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.696602 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.699146 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.699923 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.706796 (+ 6873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.707168 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.707254 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.710796 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.710800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.710803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.710835 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.712699 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.713246 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.713251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.713254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.713344 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.713746 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.713751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.713753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.713984 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.714098 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.714518 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.714654 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:25:09.715307 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.694277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.2:52433)
0619 02:25:09.695628 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.695641 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.695990 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.696806 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.696808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.697126 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.697951 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.697970 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.705692 (+ 7722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.705698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.706643 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.706651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.707427 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.707432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.707531 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.708013 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.708027 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.710674 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.712835 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.712839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.712841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.713117 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.713493 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.713496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.713498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.713628 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.714180 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.714185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.714297 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.714985 (+ 688us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.715171 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"mutex_wait_us":195,"spinlock_wait_cycles":78208,"thread_start_us":174,"threads_started":1}
I20260619 02:25:09.715968 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40883
I20260619 02:25:09.716073 7602 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:09.716257 7602 heartbeater.cc:511] Master 127.0.98.62:40883 requested a full tablet report, sending...
I20260619 02:25:09.716881 6502 ts_manager.cc:194] Re-registered known tserver with Master: 12458f15b67c479ca476b132816f5ac5 (127.0.98.2:36889)
I20260619 02:25:09.717492 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52433
I20260619 02:25:09.729475 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.725972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45005 (local address 127.0.0.1:48572)
0619 02:25:09.726101 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.726116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.726194 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.726461 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.726463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.726476 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:09.726548 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.726551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.727540 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.727542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.728270 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.728277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.729178 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.729181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.729245 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.729263 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.729314 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.729373 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 02:25:09.729559 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.726043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.0.98.60:45005)
0619 02:25:09.726162 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.726166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.726181 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:25:09.726286 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.726289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.726346 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.726448 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:09.726635 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.727433 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.728379 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.729056 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.729168 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.729316 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.729349 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.729414 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:25:09.730322 7287 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 02:25:09.730762 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8e16d7c823b4c189a05e3c8bb416c35. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } } }
I20260619 02:25:09.730906 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [sys.catalog]: This master's current role is: LEARNER
W20260619 02:25:09.823653 7608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:09.823953 7608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:09.823999 7608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:09.828137 7608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:09.828202 7608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:09.828261 7608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:25:09.833489 7608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35173
--tserver_master_addrs=127.0.98.62:40883,127.0.98.61:36913,127.0.98.60:45005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:09.834664 7608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:09.835654 7608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:09.845273 7614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:09.845250 7616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:09.845458 7608 server_base.cc:1034] running on GCE node
W20260619 02:25:09.845764 7613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:09.846180 7608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:09.846909 7608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:09.848096 7608 hybrid_clock.cc:648] HybridClock initialized: now 1781835909848078 us; error 33 us; skew 500 ppm
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:09.850894 7608 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 02:25:09.851718 7608 webserver.cc:460] Webserver started at http://127.0.98.3:35173/ using document root <none> and password file <none>
I20260619 02:25:09.852272 7608 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:09.852385 7608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:09.856266 7608 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:25:09.858418 7623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:09.859774 7608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:25:09.859920 7608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "5ba57c2beb154a38adc35cdac706fcb1"
format_stamp: "Formatted at 2026-06-19 02:25:08 on dist-test-slave-xkrt"
server_key: "d50d4a4f965d80a398b336e2205db7bf"
server_key_iv: "a6ce15a52d9388fad86bd973f7101c93"
server_key_version: "encryptionkey@0"
I20260619 02:25:09.860450 7608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:09.871160 7608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:09.874169 7608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:09.874364 7608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:09.875068 7608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:09.875979 7608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:09.876027 7608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:09.876066 7608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:09.876081 7608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:09.885939 7608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41937
I20260619 02:25:09.885999 7736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41937 every 8 connection(s)
I20260619 02:25:09.886978 7608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:25:09.895705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 7608
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 02:25:09 dist-test-slave-xkrt krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835909, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 02:25:09.904517 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.890573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43937 (local address 127.0.98.61:36913)
0619 02:25:09.890688 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.890694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.890821 (+ 127us) server_negotiation.cc:407] Connection header received
0619 02:25:09.891253 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.891257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.891309 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.891365 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.893265 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.894099 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.896254 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.896596 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.896662 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.900459 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.900463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.900465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.900490 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.902048 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.902876 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.902879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.902881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.902961 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.903328 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.903331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.903333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.903540 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.903604 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.904234 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.904357 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:25:09.904520 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.889066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45197 (local address 127.0.98.60:45005)
0619 02:25:09.889179 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.889184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.890003 (+ 819us) server_negotiation.cc:407] Connection header received
0619 02:25:09.891259 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.891263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.891318 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.891366 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.893263 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.894094 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.896254 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.896588 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.896662 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.900515 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.900518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.900521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.900554 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.902270 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.903006 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.903010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.903012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.903100 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.903394 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.903398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.903399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.903598 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.903664 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.904247 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.904365 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:25:09.904518 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.888891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38963 (local address 127.0.98.62:40883)
0619 02:25:09.889046 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.889050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.889998 (+ 948us) server_negotiation.cc:407] Connection header received
0619 02:25:09.891254 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.891257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.891309 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.891371 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.893265 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.894096 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.896039 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.896355 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.896463 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.899787 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.899791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.899793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.899820 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.902300 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.902896 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.902900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.902902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.903002 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.903366 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.903372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.903377 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.903592 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.903664 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.904247 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.904365 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:25:09.906198 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.890442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36913 (local address 127.0.98.3:43937)
0619 02:25:09.890717 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.890730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.891025 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.891736 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.891745 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.892337 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.893103 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.893122 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.894265 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.894269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.896137 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.896144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.896826 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.896829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.896977 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.898359 (+ 1382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.898373 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.900312 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.902343 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.902349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.902364 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.902746 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.903062 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.903070 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.903072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.903220 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.903676 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.903682 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.903984 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.904713 (+ 729us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.905167 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":799,"spinlock_wait_cycles":14464,"thread_start_us":105,"threads_started":1}
I20260619 02:25:09.906194 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.889347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45005 (local address 127.0.98.3:45197)
0619 02:25:09.889845 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.889897 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.891027 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.891736 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.891744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.892338 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.893107 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.893123 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.894257 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.894260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.896137 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.896144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.896813 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.896818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.896953 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.897764 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.897779 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.900048 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.902400 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.902403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.902405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.902905 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.903202 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.903205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.903206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.903293 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.903738 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.903741 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.903998 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.904719 (+ 721us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.905080 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":627,"spinlock_wait_cycles":19840,"thread_start_us":170,"threads_started":1}
I20260619 02:25:09.906194 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.889347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.3:38963)
0619 02:25:09.889845 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.889897 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.891025 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.891736 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.891744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.892337 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:09.893113 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.893122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.894265 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.894269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.895912 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.895922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.896640 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.896646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.896846 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.897503 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:09.897529 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:09.899617 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:09.902401 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.902403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.902405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.902775 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.903120 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:09.903126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:09.903130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:09.903238 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:09.903738 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:09.903741 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:09.903998 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.905148 (+ 1150us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.905257 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":490,"spinlock_wait_cycles":87040,"thread_start_us":196,"threads_started":1}
I20260619 02:25:09.907459 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.60:45005
I20260619 02:25:09.907720 7737 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:09.908303 7737 heartbeater.cc:511] Master 127.0.98.60:45005 requested a full tablet report, sending...
I20260619 02:25:09.908939 7738 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40883
I20260619 02:25:09.909026 7738 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:09.909176 7738 heartbeater.cc:511] Master 127.0.98.62:40883 requested a full tablet report, sending...
I20260619 02:25:09.907496 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36913
I20260619 02:25:09.909278 7739 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:09.909425 7739 heartbeater.cc:511] Master 127.0.98.61:36913 requested a full tablet report, sending...
I20260619 02:25:09.910271 6502 ts_manager.cc:194] Re-registered known tserver with Master: 5ba57c2beb154a38adc35cdac706fcb1 (127.0.98.3:41937)
I20260619 02:25:09.910934 7277 ts_manager.cc:194] Registered new tserver with Master: 5ba57c2beb154a38adc35cdac706fcb1 (127.0.98.3:41937)
I20260619 02:25:09.910995 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38963
I20260619 02:25:09.911577 7102 ts_manager.cc:194] Registered new tserver with Master: 5ba57c2beb154a38adc35cdac706fcb1 (127.0.98.3:41937)
I20260619 02:25:09.917819 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.906061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.0.98.61:36913)
0619 02:25:09.906165 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.906168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.907541 (+ 1373us) server_negotiation.cc:407] Connection header received
0619 02:25:09.907761 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.907768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.907819 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.907907 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.908996 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.909815 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.910943 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.911235 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.911310 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.913079 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.913082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.913087 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.913109 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.915708 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.916219 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.916223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.916224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.916304 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.916671 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.916676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.916679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.916862 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.917010 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.917530 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.917661 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:25:09.917868 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.906823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.0.98.60:45005)
0619 02:25:09.906934 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.906937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.908053 (+ 1116us) server_negotiation.cc:407] Connection header received
0619 02:25:09.908280 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.908283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.908327 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.908378 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.910115 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.910768 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.912364 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.912596 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.912708 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.914301 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.914304 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.914307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.914329 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.916001 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.916443 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.916448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.916452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.916531 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.916860 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.916864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.916866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.917108 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.917213 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.917640 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.917737 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:25:09.919625 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.907550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.0.98.62:40883)
0619 02:25:09.907645 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.907649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.908054 (+ 405us) server_negotiation.cc:407] Connection header received
0619 02:25:09.908218 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.908222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.908270 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.908337 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:09.909128 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.909890 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.913972 (+ 4082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.914342 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.914449 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.916216 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:09.916220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:09.916223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:09.916245 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:09.918031 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.918647 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.918653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.918657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.918742 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:09.918994 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:09.918997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:09.918998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:09.919154 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:09.919247 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.919396 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.919476 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:25:09.922050 6502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:25:09.925132 6502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:25:09.927484 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: true } } }
I20260619 02:25:09.927627 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:09.930745 7752 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35: attempting to promote NON_VOTER f9bc13ecd1524b5e94c4e1fe5e75a555 to VOTER
I20260619 02:25:09.931059 7752 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } }
I20260619 02:25:09.932308 7112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 FOLLOWER]: Refusing update from remote peer e8e16d7c823b4c189a05e3c8bb416c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 02:25:09.932648 7247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 02:25:09.932741 7287 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 1 LEARNER]: Refusing update from remote peer e8e16d7c823b4c189a05e3c8bb416c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 02:25:09.933113 7247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 02:25:09.934237 7247 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f9bc13ecd1524b5e94c4e1fe5e75a555 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } } }
I20260619 02:25:09.934702 7112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f9bc13ecd1524b5e94c4e1fe5e75a555 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } } }
I20260619 02:25:09.935989 7752 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } } }
I20260619 02:25:09.936100 7752 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:09.936015 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } } }
I20260619 02:25:09.936159 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:09.936218 7752 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } } }
I20260619 02:25:09.936321 7752 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:09.936662 7287 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f9bc13ecd1524b5e94c4e1fe5e75a555 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } } }
I20260619 02:25:09.937131 7761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } } }
I20260619 02:25:09.937280 7761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:25:09.938866 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } } }
I20260619 02:25:09.938972 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:25:09.945833 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.939262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41937 (local address 127.0.0.1:56244)
0619 02:25:09.940027 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.940041 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.940157 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.941025 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.941028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.941046 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:09.941174 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.941181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.943122 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.943125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.944428 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.944438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.945323 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.945327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.945513 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.945567 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.945636 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.945713 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":68,"spinlock_wait_cycles":4480}
I20260619 02:25:09.946331 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.939533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36963 (local address 127.0.0.1:40272)
0619 02:25:09.940046 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.940058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.940160 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.942464 (+ 2304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.942467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.942484 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:09.942559 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.942564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.943790 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.943794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.944879 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.944887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.945931 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.945935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.946031 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.946050 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.946125 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.946198 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"spinlock_wait_cycles":6912,"thread_start_us":84,"threads_started":1}
I20260619 02:25:09.946409 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.939699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.0.98.1:36963)
0619 02:25:09.942000 (+ 2301us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.942013 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.942030 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:25:09.942089 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.942092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.942281 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.942496 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:09.942647 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.943629 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.944997 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.945801 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.945915 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.946039 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.946160 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.946251 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2197,"thread_start_us":80,"threads_started":1}
I20260619 02:25:09.947106 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.939836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36889 (local address 127.0.0.1:50008)
0619 02:25:09.940354 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:09.940366 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:09.940436 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:09.941650 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:09.941653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:09.941676 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:09.941750 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.941754 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.943102 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.943108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.945494 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:09.945512 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.946741 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.946744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:09.946839 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.946859 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 02:25:09.946928 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 02:25:09.946989 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":40,"thread_start_us":82,"threads_started":1}
I20260619 02:25:09.947266 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.939423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.0.98.3:41937)
0619 02:25:09.939887 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.939903 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.940516 (+ 613us) server_negotiation.cc:407] Connection header received
0619 02:25:09.940603 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.940609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.940857 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.941534 (+ 677us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:09.941722 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.943005 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.944546 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.945200 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.945305 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.945446 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.946983 (+ 1537us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.947098 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":114,"threads_started":1}
I20260619 02:25:09.947499 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:09.940223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.0.98.2:36889)
0619 02:25:09.941127 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:09.941142 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:09.941165 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:25:09.941243 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:09.941248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:09.941481 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:09.941657 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:09.941859 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.942948 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.945633 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:09.946429 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:09.946550 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:09.946679 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:09.947271 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 02:25:09.947367 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":120,"threads_started":1}
I20260619 02:25:09.949443 7398 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7bc3c67c1d4a6bb8f60fbcb2a123ec (DEFAULT_TABLE table=test-table [id=d604c51b7e164214a81ad7696238f0da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:09.950628 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7bc3c67c1d4a6bb8f60fbcb2a123ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:09.950268 7535 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7bc3c67c1d4a6bb8f60fbcb2a123ec (DEFAULT_TABLE table=test-table [id=d604c51b7e164214a81ad7696238f0da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:09.951268 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7bc3c67c1d4a6bb8f60fbcb2a123ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:09.951840 7671 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7bc3c67c1d4a6bb8f60fbcb2a123ec (DEFAULT_TABLE table=test-table [id=d604c51b7e164214a81ad7696238f0da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:09.952622 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7bc3c67c1d4a6bb8f60fbcb2a123ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:09.959414 7769 tablet_bootstrap.cc:492] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1: Bootstrap starting.
I20260619 02:25:09.959415 7767 tablet_bootstrap.cc:492] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123: Bootstrap starting.
I20260619 02:25:09.961916 7768 tablet_bootstrap.cc:492] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5: Bootstrap starting.
I20260619 02:25:09.962044 7767 tablet_bootstrap.cc:654] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:09.962069 7769 tablet_bootstrap.cc:654] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:09.962821 7767 log.cc:824] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:09.962860 7769 log.cc:824] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:09.964113 7768 tablet_bootstrap.cc:654] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:09.964812 7767 tablet_bootstrap.cc:492] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123: No bootstrap required, opened a new log
I20260619 02:25:09.964819 7768 log.cc:824] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:09.965035 7767 ts_tablet_manager.cc:1397] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 02:25:09.966502 7768 tablet_bootstrap.cc:492] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5: No bootstrap required, opened a new log
I20260619 02:25:09.966686 7768 ts_tablet_manager.cc:1397] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 02:25:09.968915 7767 raft_consensus.cc:348] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } } peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } }
I20260619 02:25:09.969151 7767 raft_consensus.cc:374] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:09.969215 7767 raft_consensus.cc:729] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 990e7d22b7cd4a0baefea77792def123, State: Initialized, Role: FOLLOWER
I20260619 02:25:09.969696 7767 consensus_queue.cc:260] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } } peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } }
I20260619 02:25:09.970523 7767 ts_tablet_manager.cc:1428] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:25:09.970767 7465 heartbeater.cc:503] Master 127.0.98.62:40883 was elected leader, sending a full tablet report...
I20260619 02:25:09.972389 7769 tablet_bootstrap.cc:492] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1: No bootstrap required, opened a new log
I20260619 02:25:09.972810 7769 ts_tablet_manager.cc:1397] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20260619 02:25:09.973973 7768 raft_consensus.cc:348] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } } peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } }
I20260619 02:25:09.974251 7768 raft_consensus.cc:374] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:09.974313 7768 raft_consensus.cc:729] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12458f15b67c479ca476b132816f5ac5, State: Initialized, Role: FOLLOWER
I20260619 02:25:09.974864 7768 consensus_queue.cc:260] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } } peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } }
I20260619 02:25:09.975905 7768 ts_tablet_manager.cc:1428] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260619 02:25:09.976287 7602 heartbeater.cc:503] Master 127.0.98.62:40883 was elected leader, sending a full tablet report...
I20260619 02:25:09.977138 7769 raft_consensus.cc:348] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } } peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } }
I20260619 02:25:09.977380 7769 raft_consensus.cc:374] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:09.977439 7769 raft_consensus.cc:729] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba57c2beb154a38adc35cdac706fcb1, State: Initialized, Role: FOLLOWER
I20260619 02:25:09.977917 7769 consensus_queue.cc:260] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } } peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } }
I20260619 02:25:09.978909 7769 ts_tablet_manager.cc:1428] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:25:09.979367 7738 heartbeater.cc:503] Master 127.0.98.62:40883 was elected leader, sending a full tablet report...
W20260619 02:25:09.985628 7467 tablet.cc:2369] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:25:10.036105 7775 raft_consensus.cc:482] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:25:10.036293 7775 raft_consensus.cc:504] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } } peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } }
I20260619 02:25:10.037431 7775 leader_election.cc:288] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 990e7d22b7cd4a0baefea77792def123 (127.0.98.1:36963), 12458f15b67c479ca476b132816f5ac5 (127.0.98.2:36889)
I20260619 02:25:10.041055 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:10.037792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54815 (local address 127.0.98.1:36963)
0619 02:25:10.037923 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:10.037926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:10.037941 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:25:10.038083 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:10.038086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:10.038139 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:10.038231 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:10.038522 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.039032 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.040089 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.040489 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.040612 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:10.040674 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:10.040863 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 02:25:10.040926 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:25:10.041055 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:10.037726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36963 (local address 127.0.98.3:54815)
0619 02:25:10.037849 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:10.037866 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:10.037958 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:10.038258 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:10.038261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:10.038280 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:10.038407 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:10.038413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.039145 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.039151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:10.039960 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:10.039969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.040625 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.040629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:10.040726 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:10.040756 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 02:25:10.040849 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 02:25:10.040925 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 02:25:10.041208 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:10.038145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54651 (local address 127.0.98.2:36889)
0619 02:25:10.038306 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:10.038311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:10.038325 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:25:10.038378 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:10.038380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:10.038418 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:10.038510 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:10.038748 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.039232 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.040232 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.040670 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.040784 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:10.040840 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:10.041018 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 02:25:10.041092 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:25:10.041208 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:10.038005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36889 (local address 127.0.98.3:54651)
0619 02:25:10.038135 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:10.038145 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:10.038232 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:10.038518 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:10.038521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:10.038536 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:10.038641 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:10.038645 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.039344 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.039350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:10.040104 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:10.040112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.040784 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.040788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:10.040884 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:10.040911 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 02:25:10.041003 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:25:10.041089 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":1}
I20260619 02:25:10.041797 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7bc3c67c1d4a6bb8f60fbcb2a123ec" candidate_uuid: "5ba57c2beb154a38adc35cdac706fcb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "990e7d22b7cd4a0baefea77792def123" is_pre_election: true
I20260619 02:25:10.041903 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7bc3c67c1d4a6bb8f60fbcb2a123ec" candidate_uuid: "5ba57c2beb154a38adc35cdac706fcb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12458f15b67c479ca476b132816f5ac5" is_pre_election: true
I20260619 02:25:10.042169 7418 raft_consensus.cc:2454] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ba57c2beb154a38adc35cdac706fcb1 in term 0.
I20260619 02:25:10.042202 7555 raft_consensus.cc:2454] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ba57c2beb154a38adc35cdac706fcb1 in term 0.
I20260619 02:25:10.042793 7626 leader_election.cc:302] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba57c2beb154a38adc35cdac706fcb1, 990e7d22b7cd4a0baefea77792def123; no voters:
I20260619 02:25:10.043130 7775 raft_consensus.cc:2781] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:25:10.043239 7775 raft_consensus.cc:482] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:25:10.043315 7775 raft_consensus.cc:3037] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:10.044466 7775 raft_consensus.cc:504] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } } peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } }
I20260619 02:25:10.044850 7775 leader_election.cc:288] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [CANDIDATE]: Term 1 election: Requested vote from peers 990e7d22b7cd4a0baefea77792def123 (127.0.98.1:36963), 12458f15b67c479ca476b132816f5ac5 (127.0.98.2:36889)
I20260619 02:25:10.045262 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7bc3c67c1d4a6bb8f60fbcb2a123ec" candidate_uuid: "5ba57c2beb154a38adc35cdac706fcb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12458f15b67c479ca476b132816f5ac5"
I20260619 02:25:10.045362 7555 raft_consensus.cc:3037] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:10.045642 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7bc3c67c1d4a6bb8f60fbcb2a123ec" candidate_uuid: "5ba57c2beb154a38adc35cdac706fcb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "990e7d22b7cd4a0baefea77792def123"
I20260619 02:25:10.045785 7418 raft_consensus.cc:3037] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:10.046831 7418 raft_consensus.cc:2454] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba57c2beb154a38adc35cdac706fcb1 in term 1.
I20260619 02:25:10.047019 7555 raft_consensus.cc:2454] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba57c2beb154a38adc35cdac706fcb1 in term 1.
I20260619 02:25:10.047133 7626 leader_election.cc:302] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba57c2beb154a38adc35cdac706fcb1, 990e7d22b7cd4a0baefea77792def123; no voters:
I20260619 02:25:10.047287 7775 raft_consensus.cc:2781] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:10.047499 7775 raft_consensus.cc:686] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [term 1 LEADER]: Becoming Leader. State: Replica: 5ba57c2beb154a38adc35cdac706fcb1, State: Running, Role: LEADER
I20260619 02:25:10.047822 7775 consensus_queue.cc:237] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } } peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } }
I20260619 02:25:10.051194 6502 catalog_manager.cc:5495] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ba57c2beb154a38adc35cdac706fcb1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5ba57c2beb154a38adc35cdac706fcb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } health_report { overall_health: UNKNOWN } } }
I20260619 02:25:10.077401 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:10.073430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.0.98.3:41937)
0619 02:25:10.073604 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:10.073609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:10.073625 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:25:10.073791 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:10.073794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:10.073851 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:10.074002 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:25:10.074197 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.074777 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.075815 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.076039 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.076118 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:10.076503 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:25:10.076971 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:25:10.077074 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:10.077223 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 02:25:10.077281 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:25:10.083786 7555 raft_consensus.cc:1261] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5 [term 1 FOLLOWER]: Refusing update from remote peer 5ba57c2beb154a38adc35cdac706fcb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:10.083829 7418 raft_consensus.cc:1261] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123 [term 1 FOLLOWER]: Refusing update from remote peer 5ba57c2beb154a38adc35cdac706fcb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:10.084591 7775 consensus_queue.cc:1035] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:10.084872 7776 consensus_queue.cc:1035] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:10.093884 7779 mvcc.cc:205] Tried to move back new op lower bound from 7298399887694987264 to 7298399887558815744. Current Snapshot: MvccSnapshot[applied={T|T < 7298399887694987264}]
I20260619 02:25:10.095244 7780 mvcc.cc:205] Tried to move back new op lower bound from 7298399887694987264 to 7298399887558815744. Current Snapshot: MvccSnapshot[applied={T|T < 7298399887694987264}]
I20260619 02:25:10.104899 6501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:10.105049 6501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:10.109146 6501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8e16d7c823b4c189a05e3c8bb416c35: Sending DeleteTablet for 3 replicas of tablet 4a7bc3c67c1d4a6bb8f60fbcb2a123ec
I20260619 02:25:10.109853 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a7bc3c67c1d4a6bb8f60fbcb2a123ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56244
I20260619 02:25:10.110356 7789 tablet_replica.cc:330] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1: stopping tablet replica
I20260619 02:25:10.110351 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a7bc3c67c1d4a6bb8f60fbcb2a123ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50008
I20260619 02:25:10.110497 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a7bc3c67c1d4a6bb8f60fbcb2a123ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40272
I20260619 02:25:10.110916 7791 tablet_replica.cc:330] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123: stopping tablet replica
I20260619 02:25:10.111243 7791 raft_consensus.cc:2229] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:10.111270 7789 raft_consensus.cc:2229] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:25:10.111750 7789 raft_consensus.cc:2258] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:10.111977 7791 raft_consensus.cc:2258] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:10.113399 7789 ts_tablet_manager.cc:1905] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:10.113749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 6470
I20260619 02:25:10.114668 7791 ts_tablet_manager.cc:1905] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:10.117401 7789 ts_tablet_manager.cc:1918] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:10.117589 7789 log.cc:1196] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/4a7bc3c67c1d4a6bb8f60fbcb2a123ec
I20260619 02:25:10.117985 7789 ts_tablet_manager.cc:1939] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 5ba57c2beb154a38adc35cdac706fcb1: Deleting consensus metadata
I20260619 02:25:10.118142 7791 ts_tablet_manager.cc:1918] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:10.118245 7791 log.cc:1196] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-0/wal/wals/4a7bc3c67c1d4a6bb8f60fbcb2a123ec
I20260619 02:25:10.118569 7791 ts_tablet_manager.cc:1939] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 990e7d22b7cd4a0baefea77792def123: Deleting consensus metadata
I20260619 02:25:10.119896 7790 tablet_replica.cc:330] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5: stopping tablet replica
I20260619 02:25:10.120260 7790 raft_consensus.cc:2229] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:10.120626 7790 raft_consensus.cc:2258] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 02:25:10.121650 7624 connection.cc:535] server connection from 127.0.0.1:56244 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:25:10.122263 7790 ts_tablet_manager.cc:1905] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 02:25:10.122545 7351 connection.cc:535] server connection from 127.0.0.1:40272 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:25:10.125057 7790 ts_tablet_manager.cc:1918] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:10.125197 7790 log.cc:1196] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/4a7bc3c67c1d4a6bb8f60fbcb2a123ec
I20260619 02:25:10.125609 7790 ts_tablet_manager.cc:1939] T 4a7bc3c67c1d4a6bb8f60fbcb2a123ec P 12458f15b67c479ca476b132816f5ac5: Deleting consensus metadata
W20260619 02:25:10.127014 7491 connection.cc:441] server connection from 127.0.0.1:50008 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50008 (request call id 1) could send its response
I20260619 02:25:10.130625 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:10.112973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.0.98.61:36913)
0619 02:25:10.113070 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:10.113075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:10.119683 (+ 6608us) server_negotiation.cc:407] Connection header received
0619 02:25:10.119740 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:10.119746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:10.119811 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:10.119892 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:10.121168 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.122018 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.124067 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.124400 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.124479 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:10.126474 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:10.126478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:10.126482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:10.126512 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:10.128653 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:10.129134 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:10.129139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:10.129140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:10.129226 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:10.129574 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:10.129577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:10.129579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:10.129788 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:10.129902 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:10.130253 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 02:25:10.130463 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:25:10.137439 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:10.125980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.0.98.60:45005)
0619 02:25:10.126478 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:10.126482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:10.127729 (+ 1247us) server_negotiation.cc:407] Connection header received
0619 02:25:10.127822 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:10.127826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:10.127880 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:10.129034 (+ 1154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:10.129162 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.129955 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.131094 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.131468 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.131909 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:10.133162 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:10.133166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:10.133169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:10.133193 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:10.135208 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:10.135649 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:10.135652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:10.135654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:10.135738 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:10.136056 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:10.136061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:10.136064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:10.136295 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:10.137024 (+ 729us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:10.137101 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 02:25:10.137225 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":424,"thread_start_us":88,"threads_started":1}
I20260619 02:25:10.137477 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:10.125294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52730 (local address 127.0.98.61:36913)
0619 02:25:10.126785 (+ 1491us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:10.126796 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:10.126810 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:25:10.126842 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:10.126845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:10.126885 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:10.126976 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:10.129181 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.130018 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.131472 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.131848 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.131919 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:10.134075 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:10.134079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:10.134081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:10.134105 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:10.135763 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:10.136229 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:10.136233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:10.136235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:10.136313 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:10.136585 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:10.136588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:10.136590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:10.136784 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:10.136882 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:10.137196 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 02:25:10.137310 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1412,"thread_start_us":85,"threads_started":1}
I20260619 02:25:10.140419 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:10.112480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48578 (local address 127.0.98.60:45005)
0619 02:25:10.112590 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:10.112595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:10.131261 (+ 18666us) server_negotiation.cc:407] Connection header received
0619 02:25:10.131306 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:10.131308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:10.131348 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:10.131411 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:10.133087 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.133921 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.135222 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:10.135573 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:10.135662 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:10.137210 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:10.137213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:10.137216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:10.137235 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:10.138822 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:10.139270 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:10.139273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:10.139275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:10.139353 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:10.139620 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:10.139623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:10.139625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:10.139821 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:10.139938 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:10.140170 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 02:25:10.140271 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":648,"server-negotiator.queue_time_us":40}
I20260619 02:25:11.098724 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.097383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.3:56945)
0619 02:25:11.098505 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.098600 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1036,"thread_start_us":172,"threads_started":1}
W20260619 02:25:11.099167 7738 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40883 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
I20260619 02:25:11.099313 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.098588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.1:49805)
0619 02:25:11.099133 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.099214 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":55,"threads_started":1}
I20260619 02:25:11.099572 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.097883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.2:37149)
0619 02:25:11.098548 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.098647 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":64,"threads_started":1}
W20260619 02:25:11.099603 7465 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40883 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
W20260619 02:25:11.099833 7602 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40883 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
I20260619 02:25:11.609220 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e8e16d7c823b4c189a05e3c8bb416c35)
I20260619 02:25:11.609445 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } }
I20260619 02:25:11.610226 7806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e8e16d7c823b4c189a05e3c8bb416c35)
I20260619 02:25:11.610476 7806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } }
I20260619 02:25:11.610618 7805 leader_election.cc:288] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8e16d7c823b4c189a05e3c8bb416c35 (127.0.98.62:40883), 53fc572233564e31945ad52c925ba385 (127.0.98.61:36913)
I20260619 02:25:11.611485 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.610791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.0.1:55520)
0619 02:25:11.611201 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.611283 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":86,"threads_started":1}
W20260619 02:25:11.611675 7269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
I20260619 02:25:11.611812 7806 leader_election.cc:288] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8e16d7c823b4c189a05e3c8bb416c35 (127.0.98.62:40883), f9bc13ecd1524b5e94c4e1fe5e75a555 (127.0.98.60:45005)
I20260619 02:25:11.612150 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.611538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.0.1:55524)
0619 02:25:11.612002 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.612066 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":66,"threads_started":1}
W20260619 02:25:11.612288 7086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
I20260619 02:25:11.613099 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.612840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.0.1:55540)
0619 02:25:11.612989 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.613037 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 02:25:11.613178 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.612918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.0.1:55552)
0619 02:25:11.613057 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.613105 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260619 02:25:11.613235 7269 leader_election.cc:334] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e8e16d7c823b4c189a05e3c8bb416c35 (127.0.98.62:40883): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
W20260619 02:25:11.613320 7086 leader_election.cc:334] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e8e16d7c823b4c189a05e3c8bb416c35 (127.0.98.62:40883): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
I20260619 02:25:11.615273 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.610883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36913 (local address 127.0.0.1:52744)
0619 02:25:11.611260 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.611272 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:11.611424 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:11.611801 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:11.611805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:11.611829 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:11.611950 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:11.611956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.612989 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.612992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:11.614107 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:11.614119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.614883 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.614887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:11.614972 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:11.614997 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:25:11.615079 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 02:25:11.615167 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260619 02:25:11.615293 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.610927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.0.98.61:36913)
0619 02:25:11.611185 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:11.611190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:11.611366 (+ 176us) server_negotiation.cc:407] Connection header received
0619 02:25:11.611546 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:11.611551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:11.611604 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:11.611680 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:11.612075 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.612870 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.614226 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.614761 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.614871 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:11.614926 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:11.615097 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 02:25:11.615184 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":54,"threads_started":1}
I20260619 02:25:11.615746 7112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53fc572233564e31945ad52c925ba385" is_pre_election: true
I20260619 02:25:11.615964 7112 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9bc13ecd1524b5e94c4e1fe5e75a555 in term 1.
I20260619 02:25:11.616253 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.612021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45005 (local address 127.0.0.1:48602)
0619 02:25:11.612338 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.612352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:11.612441 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:11.612876 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:11.612880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:11.612896 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:11.612983 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:11.612992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.614133 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.614136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:11.615066 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:11.615078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.615858 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.615862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:11.615948 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:11.615970 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:25:11.616062 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:25:11.616129 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":52,"threads_started":1}
I20260619 02:25:11.616288 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.612333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.0.98.60:45005)
0619 02:25:11.612627 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:11.612636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:11.612650 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:25:11.612691 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:11.612694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:11.612736 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:11.613089 (+ 353us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:11.613491 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.614030 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.615178 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.615748 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.615836 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:11.615905 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:11.616098 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 02:25:11.616168 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":224,"server-negotiator.queue_time_us":232,"thread_start_us":87,"threads_started":1}
I20260619 02:25:11.616313 7270 leader_election.cc:302] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53fc572233564e31945ad52c925ba385, f9bc13ecd1524b5e94c4e1fe5e75a555; no voters: e8e16d7c823b4c189a05e3c8bb416c35
I20260619 02:25:11.616578 7805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 02:25:11.616664 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e8e16d7c823b4c189a05e3c8bb416c35)
I20260619 02:25:11.616719 7805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:25:11.616927 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53fc572233564e31945ad52c925ba385" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" is_pre_election: true
I20260619 02:25:11.617867 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } }
I20260619 02:25:11.618111 7287 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 53fc572233564e31945ad52c925ba385 in current term 2: Already voted for candidate f9bc13ecd1524b5e94c4e1fe5e75a555 in this term.
I20260619 02:25:11.618261 7805 leader_election.cc:288] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [CANDIDATE]: Term 2 election: Requested vote from peers e8e16d7c823b4c189a05e3c8bb416c35 (127.0.98.62:40883), 53fc572233564e31945ad52c925ba385 (127.0.98.61:36913)
I20260619 02:25:11.618547 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.618335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.0.1:55562)
0619 02:25:11.618414 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.618490 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 02:25:11.618780 7112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53fc572233564e31945ad52c925ba385"
I20260619 02:25:11.618896 7112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:25:11.619061 7087 leader_election.cc:302] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53fc572233564e31945ad52c925ba385; no voters: e8e16d7c823b4c189a05e3c8bb416c35, f9bc13ecd1524b5e94c4e1fe5e75a555
I20260619 02:25:11.619144 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.618940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.0.1:55566)
0619 02:25:11.619033 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.619088 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 02:25:11.619256 7269 leader_election.cc:334] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e8e16d7c823b4c189a05e3c8bb416c35 (127.0.98.62:40883): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
I20260619 02:25:11.619899 7112 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9bc13ecd1524b5e94c4e1fe5e75a555 in term 2.
I20260619 02:25:11.620018 7806 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:25:11.620198 7270 leader_election.cc:302] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53fc572233564e31945ad52c925ba385, f9bc13ecd1524b5e94c4e1fe5e75a555; no voters: e8e16d7c823b4c189a05e3c8bb416c35
I20260619 02:25:11.620383 7805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:25:11.620618 7805 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [term 2 LEADER]: Becoming Leader. State: Replica: f9bc13ecd1524b5e94c4e1fe5e75a555, State: Running, Role: LEADER
I20260619 02:25:11.620860 7805 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } }
I20260619 02:25:11.622032 7815 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9bc13ecd1524b5e94c4e1fe5e75a555. Latest consensus state: current_term: 2 leader_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } } }
I20260619 02:25:11.622149 7815 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:11.622462 7817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:11.624879 7817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:11.625308 7817 catalog_manager.cc:1259] Loaded cluster ID: 9d73f6d34c9c4fa8aeb90a06b2f1baa6
I20260619 02:25:11.625352 7817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:11.625952 7817 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:11.626348 7817 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: Loaded TSK: 0
I20260619 02:25:11.627027 7817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:25:11.656149 7285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:25:11.659358 7285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:25:11.662287 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.662000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.0.1:55582)
0619 02:25:11.662115 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.662209 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 02:25:11.662343 7112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [term 2 FOLLOWER]: Refusing update from remote peer f9bc13ecd1524b5e94c4e1fe5e75a555: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 02:25:11.662938 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.662687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.0.1:55596)
0619 02:25:11.662794 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.662845 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 02:25:11.662859 7815 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 02:25:11.663069 7269 consensus_peers.cc:487] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 -> Peer e8e16d7c823b4c189a05e3c8bb416c35 (127.0.98.62:40883): Couldn't send request to peer e8e16d7c823b4c189a05e3c8bb416c35. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 02:25:11.664443 7806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9bc13ecd1524b5e94c4e1fe5e75a555. Latest consensus state: current_term: 2 leader_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } } }
I20260619 02:25:11.664564 7806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:25:11.665527 7815 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } } }
I20260619 02:25:11.665663 7815 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:11.666059 7806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8e16d7c823b4c189a05e3c8bb416c35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } peers { permanent_uuid: "53fc572233564e31945ad52c925ba385" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36913 } attrs { promote: false } } peers { permanent_uuid: "f9bc13ecd1524b5e94c4e1fe5e75a555" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45005 } attrs { promote: false } } }
I20260619 02:25:11.666173 7806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53fc572233564e31945ad52c925ba385 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:25:11.666273 7819 mvcc.cc:205] Tried to move back new op lower bound from 7298399894164008960 to 7298399894001115136. Current Snapshot: MvccSnapshot[applied={T|T < 7298399894164008960 or (T in {7298399894164008960})}]
I20260619 02:25:11.675981 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.672085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50018 (local address 127.0.98.2:36889)
0619 02:25:11.672338 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:11.672342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:11.672355 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:25:11.672408 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:11.672410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:11.672447 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:11.672537 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:11.672779 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.673423 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.674849 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.675451 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.675568 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:11.675634 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:11.675780 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 02:25:11.675852 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260619 02:25:11.675980 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.671650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36889 (local address 127.0.0.1:50018)
0619 02:25:11.671779 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.671790 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:11.671878 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:11.672543 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:11.672546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:11.672562 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:11.672656 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:11.672664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.673523 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.673529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:11.674667 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:11.674677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.675563 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.675567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:11.675646 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:11.675680 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 02:25:11.675766 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:25:11.675846 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":3,"spinlock_wait_cycles":7424}
I20260619 02:25:11.676352 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.671446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36963 (local address 127.0.0.1:40278)
0619 02:25:11.671764 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.671779 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:11.671871 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:11.672767 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:11.672773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:11.672791 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:11.672892 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:11.672898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.673964 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.673968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:11.675145 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:11.675154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.675987 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.675992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:11.676087 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:11.676120 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 02:25:11.676177 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 02:25:11.676237 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":37}
I20260619 02:25:11.676611 7535 tablet_service.cc:1467] Processing CreateTablet for tablet 399b33c6eab74eefb575c3d5198c032c (DEFAULT_TABLE table=test-table [id=3f90de70e9dd4bc1abf31e4179202cd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:11.676839 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.671535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.0.98.1:36963)
0619 02:25:11.672530 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:11.672535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:11.672547 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:25:11.672593 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:11.672596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:11.672649 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:11.673002 (+ 353us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:11.673107 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.673862 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.675240 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.675850 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.676536 (+ 686us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:11.676602 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:11.676639 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 02:25:11.676711 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":88,"threads_started":1}
I20260619 02:25:11.676860 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399b33c6eab74eefb575c3d5198c032c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:11.677155 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.671943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41937 (local address 127.0.0.1:56274)
0619 02:25:11.672166 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.672179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:11.672260 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:11.672705 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:11.672710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:11.672729 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:11.672813 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:11.672820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.673876 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.673879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:11.675063 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:11.675074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.676027 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.676030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:11.676570 (+ 540us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:11.676590 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 02:25:11.676659 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 02:25:11.677069 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":37,"thread_start_us":91,"threads_started":1}
I20260619 02:25:11.677346 7398 tablet_service.cc:1467] Processing CreateTablet for tablet 399b33c6eab74eefb575c3d5198c032c (DEFAULT_TABLE table=test-table [id=3f90de70e9dd4bc1abf31e4179202cd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:11.677490 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.672257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.0.98.3:41937)
0619 02:25:11.672466 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:11.672471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:11.672485 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:25:11.672536 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:11.672540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:11.672590 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:11.672678 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:11.672926 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.673752 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.675167 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:11.675791 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:11.675907 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:11.675981 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:11.676774 (+ 793us) server_negotiation.cc:299] Negotiation successful
0619 02:25:11.676855 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20260619 02:25:11.677626 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399b33c6eab74eefb575c3d5198c032c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:11.679517 7828 tablet_bootstrap.cc:492] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5: Bootstrap starting.
I20260619 02:25:11.680024 7829 tablet_bootstrap.cc:492] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123: Bootstrap starting.
I20260619 02:25:11.680225 7671 tablet_service.cc:1467] Processing CreateTablet for tablet 399b33c6eab74eefb575c3d5198c032c (DEFAULT_TABLE table=test-table [id=3f90de70e9dd4bc1abf31e4179202cd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:11.680389 7828 tablet_bootstrap.cc:654] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:11.680485 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399b33c6eab74eefb575c3d5198c032c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:11.680857 7829 tablet_bootstrap.cc:654] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:11.681694 7828 tablet_bootstrap.cc:492] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5: No bootstrap required, opened a new log
I20260619 02:25:11.681792 7828 ts_tablet_manager.cc:1397] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:11.682287 7828 raft_consensus.cc:348] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } } peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } }
I20260619 02:25:11.682392 7829 tablet_bootstrap.cc:492] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123: No bootstrap required, opened a new log
I20260619 02:25:11.682416 7828 raft_consensus.cc:374] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:11.682456 7828 raft_consensus.cc:729] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12458f15b67c479ca476b132816f5ac5, State: Initialized, Role: FOLLOWER
I20260619 02:25:11.682464 7829 ts_tablet_manager.cc:1397] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:25:11.682608 7828 consensus_queue.cc:260] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } } peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } }
I20260619 02:25:11.682945 7829 raft_consensus.cc:348] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } } peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } }
I20260619 02:25:11.683066 7829 raft_consensus.cc:374] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:11.683104 7829 raft_consensus.cc:729] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 990e7d22b7cd4a0baefea77792def123, State: Initialized, Role: FOLLOWER
I20260619 02:25:11.683239 7829 consensus_queue.cc:260] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } } peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } }
I20260619 02:25:11.683602 7829 ts_tablet_manager.cc:1428] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:25:11.683641 7828 ts_tablet_manager.cc:1428] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:25:11.684403 7833 tablet_bootstrap.cc:492] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1: Bootstrap starting.
I20260619 02:25:11.685286 7833 tablet_bootstrap.cc:654] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:11.686185 7464 heartbeater.cc:503] Master 127.0.98.60:45005 was elected leader, sending a full tablet report...
I20260619 02:25:11.686370 7601 heartbeater.cc:503] Master 127.0.98.60:45005 was elected leader, sending a full tablet report...
I20260619 02:25:11.685402 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.683359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.2:37873)
0619 02:25:11.684956 (+ 1597us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.685016 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1514,"thread_start_us":92,"threads_started":1}
I20260619 02:25:11.687413 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.686986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.2:48197)
0619 02:25:11.687251 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.687327 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
I20260619 02:25:11.687724 7833 tablet_bootstrap.cc:492] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1: No bootstrap required, opened a new log
I20260619 02:25:11.687832 7833 ts_tablet_manager.cc:1397] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 02:25:11.687889 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.683998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.1:37847)
0619 02:25:11.687761 (+ 3763us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.687816 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3701,"thread_start_us":83,"threads_started":1}
I20260619 02:25:11.688228 7833 raft_consensus.cc:348] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } } peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } }
I20260619 02:25:11.688336 7833 raft_consensus.cc:374] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:11.688376 7833 raft_consensus.cc:729] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba57c2beb154a38adc35cdac706fcb1, State: Initialized, Role: FOLLOWER
I20260619 02:25:11.688503 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.688265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.1:59339)
0619 02:25:11.688380 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.688423 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 02:25:11.688517 7833 consensus_queue.cc:260] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } } peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } }
I20260619 02:25:11.688808 7833 ts_tablet_manager.cc:1428] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:25:11.689491 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.689114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.3:52603)
0619 02:25:11.689348 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.689405 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260619 02:25:11.690156 7737 heartbeater.cc:503] Master 127.0.98.60:45005 was elected leader, sending a full tablet report...
I20260619 02:25:11.690171 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:11.689907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.3:53107)
0619 02:25:11.690045 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:11.690091 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260619 02:25:11.695145 7604 tablet.cc:2369] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:25:11.890436 7740 tablet.cc:2369] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:25:12.051465 7834 raft_consensus.cc:482] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:25:12.051620 7834 raft_consensus.cc:504] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } } peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } }
I20260619 02:25:12.052758 7834 leader_election.cc:288] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12458f15b67c479ca476b132816f5ac5 (127.0.98.2:36889), 5ba57c2beb154a38adc35cdac706fcb1 (127.0.98.3:41937)
I20260619 02:25:12.055753 7832 raft_consensus.cc:482] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:25:12.055900 7832 raft_consensus.cc:504] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } } peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } }
I20260619 02:25:12.056993 7832 leader_election.cc:288] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 990e7d22b7cd4a0baefea77792def123 (127.0.98.1:36963), 5ba57c2beb154a38adc35cdac706fcb1 (127.0.98.3:41937)
I20260619 02:25:12.057734 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.053141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50283 (local address 127.0.98.2:36889)
0619 02:25:12.053277 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:12.053281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:12.053679 (+ 398us) server_negotiation.cc:407] Connection header received
0619 02:25:12.053805 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:12.053807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:12.053849 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:12.053936 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:12.054353 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.054876 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.056301 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.056909 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.057023 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:12.057088 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:12.057484 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 02:25:12.057601 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:25:12.057987 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.053009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41937 (local address 127.0.98.1:34233)
0619 02:25:12.053141 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:12.053158 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:12.053280 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:12.053606 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:12.053611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:12.053634 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:12.053775 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:12.053782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.054886 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.054894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:12.056161 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:12.056172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.057173 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.057177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:12.057645 (+ 468us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:12.057669 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:25:12.057738 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 02:25:12.057826 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":429}
I20260619 02:25:12.058410 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.053114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34233 (local address 127.0.98.3:41937)
0619 02:25:12.053266 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:12.053273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:12.053295 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:25:12.053393 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:12.053398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:12.053453 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:12.053534 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:12.053897 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.054756 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.056346 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.056945 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.057033 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:12.057088 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:12.058233 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0619 02:25:12.058306 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:25:12.058419 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.053009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36889 (local address 127.0.98.1:50283)
0619 02:25:12.053606 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:12.053618 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:12.053695 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:12.053948 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:12.053951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:12.053968 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:12.054080 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:12.054085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.054999 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.055002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:12.056175 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:12.056185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.057185 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.057187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:12.057329 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:12.057355 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 02:25:12.058200 (+ 845us) client_negotiation.cc:240] Negotiation successful
0619 02:25:12.058270 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":231,"thread_start_us":191,"threads_started":1}
I20260619 02:25:12.058921 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399b33c6eab74eefb575c3d5198c032c" candidate_uuid: "990e7d22b7cd4a0baefea77792def123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba57c2beb154a38adc35cdac706fcb1" is_pre_election: true
I20260619 02:25:12.059187 7691 raft_consensus.cc:2454] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 990e7d22b7cd4a0baefea77792def123 in term 0.
I20260619 02:25:12.059623 7351 leader_election.cc:302] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba57c2beb154a38adc35cdac706fcb1, 990e7d22b7cd4a0baefea77792def123; no voters:
I20260619 02:25:12.059725 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399b33c6eab74eefb575c3d5198c032c" candidate_uuid: "990e7d22b7cd4a0baefea77792def123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12458f15b67c479ca476b132816f5ac5" is_pre_election: true
I20260619 02:25:12.059855 7555 raft_consensus.cc:2454] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 990e7d22b7cd4a0baefea77792def123 in term 0.
I20260619 02:25:12.059978 7834 raft_consensus.cc:2781] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:25:12.060063 7834 raft_consensus.cc:482] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:25:12.060108 7834 raft_consensus.cc:3037] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:12.061472 7834 raft_consensus.cc:504] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } } peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } }
I20260619 02:25:12.061852 7834 leader_election.cc:288] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [CANDIDATE]: Term 1 election: Requested vote from peers 12458f15b67c479ca476b132816f5ac5 (127.0.98.2:36889), 5ba57c2beb154a38adc35cdac706fcb1 (127.0.98.3:41937)
I20260619 02:25:12.062144 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.057261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41937 (local address 127.0.98.2:55077)
0619 02:25:12.057386 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:12.057401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:12.057500 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:12.058172 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:12.058175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:12.058194 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:12.058470 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:12.058480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.059530 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.059535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:12.060912 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:12.060922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.061698 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.061701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:12.061796 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:12.061844 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 02:25:12.061922 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:25:12.061996 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":167,"spinlock_wait_cycles":11008}
I20260619 02:25:12.062144 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399b33c6eab74eefb575c3d5198c032c" candidate_uuid: "990e7d22b7cd4a0baefea77792def123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12458f15b67c479ca476b132816f5ac5"
I20260619 02:25:12.062288 7555 raft_consensus.cc:3037] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:12.062273 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399b33c6eab74eefb575c3d5198c032c" candidate_uuid: "990e7d22b7cd4a0baefea77792def123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba57c2beb154a38adc35cdac706fcb1"
I20260619 02:25:12.062397 7691 raft_consensus.cc:3037] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:12.062803 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.057369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55077 (local address 127.0.98.3:41937)
0619 02:25:12.057921 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:12.057930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:12.057943 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:25:12.057987 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:12.057993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:12.058043 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:12.058319 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:12.058596 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.059407 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.061067 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.061479 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.061642 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:12.061698 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:12.062622 (+ 924us) server_negotiation.cc:299] Negotiation successful
0619 02:25:12.062691 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":114,"threads_started":1}
I20260619 02:25:12.063457 7555 raft_consensus.cc:2454] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 990e7d22b7cd4a0baefea77792def123 in term 1.
I20260619 02:25:12.063633 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.057134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36963 (local address 127.0.98.2:38019)
0619 02:25:12.057386 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:12.057401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:12.057500 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:12.058162 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:12.058165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:12.058184 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:12.058366 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:12.058374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.059494 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.059501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:12.061985 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:12.061997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.063113 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.063117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:12.063217 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:12.063241 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:25:12.063337 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 02:25:12.063437 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":1221,"spinlock_wait_cycles":2176}
I20260619 02:25:12.063735 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.057680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38019 (local address 127.0.98.1:36963)
0619 02:25:12.057837 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:12.057845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:12.057858 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:25:12.057899 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:12.057902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:12.057951 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:12.058028 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:12.058521 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.059353 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.062123 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.062745 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.062860 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:12.062914 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:12.063512 (+ 598us) server_negotiation.cc:299] Negotiation successful
0619 02:25:12.063584 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:25:12.063781 7691 raft_consensus.cc:2454] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 990e7d22b7cd4a0baefea77792def123 in term 1.
I20260619 02:25:12.063891 7690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399b33c6eab74eefb575c3d5198c032c" candidate_uuid: "12458f15b67c479ca476b132816f5ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba57c2beb154a38adc35cdac706fcb1" is_pre_election: true
I20260619 02:25:12.064085 7690 raft_consensus.cc:2379] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 12458f15b67c479ca476b132816f5ac5 in current term 1: Already voted for candidate 990e7d22b7cd4a0baefea77792def123 in this term.
I20260619 02:25:12.064072 7351 leader_election.cc:302] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba57c2beb154a38adc35cdac706fcb1, 990e7d22b7cd4a0baefea77792def123; no voters:
I20260619 02:25:12.064316 7834 raft_consensus.cc:2781] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:12.064396 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399b33c6eab74eefb575c3d5198c032c" candidate_uuid: "12458f15b67c479ca476b132816f5ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "990e7d22b7cd4a0baefea77792def123" is_pre_election: true
I20260619 02:25:12.064534 7834 raft_consensus.cc:686] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [term 1 LEADER]: Becoming Leader. State: Replica: 990e7d22b7cd4a0baefea77792def123, State: Running, Role: LEADER
I20260619 02:25:12.065059 7834 consensus_queue.cc:237] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } } peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } }
I20260619 02:25:12.065261 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.064998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.1:43017)
0619 02:25:12.065130 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:12.065189 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 02:25:12.066892 7490 leader_election.cc:302] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12458f15b67c479ca476b132816f5ac5; no voters: 5ba57c2beb154a38adc35cdac706fcb1, 990e7d22b7cd4a0baefea77792def123
I20260619 02:25:12.067165 7832 raft_consensus.cc:2726] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 02:25:12.067713 7277 catalog_manager.cc:5495] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 reported cstate change: term changed from 0 to 1, leader changed from <none> to 990e7d22b7cd4a0baefea77792def123 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "990e7d22b7cd4a0baefea77792def123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "990e7d22b7cd4a0baefea77792def123" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20260619 02:25:12.126514 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.122670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.0.98.1:36963)
0619 02:25:12.122835 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:12.122841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:12.122857 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:25:12.122944 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:12.122947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:12.123043 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:12.123185 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:25:12.123453 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.124038 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.125076 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:12.125287 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:12.125401 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:12.125673 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:25:12.126114 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:25:12.126201 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:12.126336 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 02:25:12.126383 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:25:12.129405 7555 raft_consensus.cc:1261] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [term 1 FOLLOWER]: Refusing update from remote peer 990e7d22b7cd4a0baefea77792def123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:12.129609 7690 raft_consensus.cc:1261] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1 [term 1 FOLLOWER]: Refusing update from remote peer 990e7d22b7cd4a0baefea77792def123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:12.130098 7843 consensus_queue.cc:1035] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12458f15b67c479ca476b132816f5ac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:12.130306 7834 consensus_queue.cc:1035] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba57c2beb154a38adc35cdac706fcb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:12.131948 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.131652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.2:45785)
0619 02:25:12.131800 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:12.131868 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 02:25:12.135025 7847 mvcc.cc:205] Tried to move back new op lower bound from 7298399896076615680 to 7298399895821541376. Current Snapshot: MvccSnapshot[applied={T|T < 7298399896076615680 or (T in {7298399896076615680})}]
I20260619 02:25:12.135900 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.135587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.2:53975)
0619 02:25:12.135747 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:12.135829 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 02:25:12.136399 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.135047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.1:55047)
0619 02:25:12.136267 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:12.136326 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1152}
I20260619 02:25:12.136745 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.135850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.3:55175)
0619 02:25:12.136613 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:12.136669 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":686}
I20260619 02:25:12.139536 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.139277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.3:56639)
0619 02:25:12.139423 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:12.139470 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 02:25:12.141826 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.140375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.1:56493)
0619 02:25:12.141670 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:12.141714 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1226}
I20260619 02:25:12.145850 7277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:12.146014 7277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:12.149820 7277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9bc13ecd1524b5e94c4e1fe5e75a555: Sending DeleteTablet for 3 replicas of tablet 399b33c6eab74eefb575c3d5198c032c
I20260619 02:25:12.150415 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet 399b33c6eab74eefb575c3d5198c032c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:12 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40278
I20260619 02:25:12.150843 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet 399b33c6eab74eefb575c3d5198c032c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:12 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50018
I20260619 02:25:12.150910 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet 399b33c6eab74eefb575c3d5198c032c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:12 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:56274
I20260619 02:25:12.150810 7856 tablet_replica.cc:330] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123: stopping tablet replica
I20260619 02:25:12.151077 7856 raft_consensus.cc:2229] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:25:12.151324 7857 tablet_replica.cc:330] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5: stopping tablet replica
I20260619 02:25:12.151381 7856 raft_consensus.cc:2258] T 399b33c6eab74eefb575c3d5198c032c P 990e7d22b7cd4a0baefea77792def123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:12.151672 7857 raft_consensus.cc:2229] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:12.151852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 7246
I20260619 02:25:12.153074 7857 raft_consensus.cc:2258] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:12.153342 7858 tablet_replica.cc:330] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1: stopping tablet replica
I20260619 02:25:12.153468 7858 raft_consensus.cc:2229] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:12.153645 7858 raft_consensus.cc:2258] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:12.154071 7857 ts_tablet_manager.cc:1905] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:12.154373 7858 ts_tablet_manager.cc:1905] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:12.156713 7857 ts_tablet_manager.cc:1918] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:12.156798 7857 log.cc:1196] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/399b33c6eab74eefb575c3d5198c032c
I20260619 02:25:12.156798 7858 ts_tablet_manager.cc:1918] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:12.156860 7858 log.cc:1196] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/399b33c6eab74eefb575c3d5198c032c
I20260619 02:25:12.157100 7857 ts_tablet_manager.cc:1939] T 399b33c6eab74eefb575c3d5198c032c P 12458f15b67c479ca476b132816f5ac5: Deleting consensus metadata
I20260619 02:25:12.157138 7858 ts_tablet_manager.cc:1939] T 399b33c6eab74eefb575c3d5198c032c P 5ba57c2beb154a38adc35cdac706fcb1: Deleting consensus metadata
I20260619 02:25:12.157780 7269 catalog_manager.cc:4841] TS 5ba57c2beb154a38adc35cdac706fcb1 (127.0.98.3:41937): tablet 399b33c6eab74eefb575c3d5198c032c (table test-table [id=3f90de70e9dd4bc1abf31e4179202cd4]) successfully deleted
I20260619 02:25:12.157838 7271 catalog_manager.cc:4841] TS 12458f15b67c479ca476b132816f5ac5 (127.0.98.2:36889): tablet 399b33c6eab74eefb575c3d5198c032c (table test-table [id=3f90de70e9dd4bc1abf31e4179202cd4]) successfully deleted
I20260619 02:25:12.159067 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 7472
I20260619 02:25:12.159335 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:12.159059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36963 (local address 127.0.0.1:40290)
0619 02:25:12.159188 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:12.159247 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:25:12.159914 7271 catalog_manager.cc:4584] TS 990e7d22b7cd4a0baefea77792def123 (127.0.98.1:36963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 399b33c6eab74eefb575c3d5198c032c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36963: connect: Connection refused (error 111)
I20260619 02:25:12.165783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 7608
I20260619 02:25:12.172552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 6966
I20260619 02:25:12.178498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 7074
2026-06-19T02:25:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5639 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:25:12 dist-test-slave-xkrt krb5kdc[7861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:25:12 dist-test-slave-xkrt krb5kdc[7861](info): set up 2 sockets
Jun 19 02:25:12 dist-test-slave-xkrt krb5kdc[7861](info): commencing operation
krb5kdc: starting...
W20260619 02:25:13.226658 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:25:13 dist-test-slave-xkrt krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:25:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 02:25:13.352056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:13.461031 7874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:13.461275 7874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:13.461314 7874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:13.465317 7874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:13.465380 7874 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:25:13.465399 7874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:13.465417 7874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:13.465433 7874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:13.465492 7874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:25:13.470738 7874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36889
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:13.471788 7874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:13.472808 7874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:13.481297 7879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:13.481361 7874 server_base.cc:1034] running on GCE node
W20260619 02:25:13.481313 7882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:13.481320 7880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:13.482226 7874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:13.483269 7874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:13.484465 7874 hybrid_clock.cc:648] HybridClock initialized: now 1781835913484363 us; error 111 us; skew 500 ppm
Jun 19 02:25:13 dist-test-slave-xkrt krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835913, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:13.487519 7874 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 02:25:13.488385 7874 webserver.cc:460] Webserver started at http://127.0.98.62:38087/ using document root <none> and password file <none>
I20260619 02:25:13.488952 7874 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:13.489044 7874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:13.489269 7874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:13.491375 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "5c3b5ade642d4009bc7c1e9b97ed1c2a"
format_stamp: "Formatted at 2026-06-19 02:25:13 on dist-test-slave-xkrt"
server_key: "b402cb07926fab0f0545c7e68bb776dd"
server_key_iv: "0230a547570133df6014d45f09081774"
server_key_version: "encryptionkey@0"
I20260619 02:25:13.491955 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c3b5ade642d4009bc7c1e9b97ed1c2a"
format_stamp: "Formatted at 2026-06-19 02:25:13 on dist-test-slave-xkrt"
server_key: "b402cb07926fab0f0545c7e68bb776dd"
server_key_iv: "0230a547570133df6014d45f09081774"
server_key_version: "encryptionkey@0"
I20260619 02:25:13.495642 7874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:25:13.498072 7889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:13.499467 7874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:13.499636 7874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "5c3b5ade642d4009bc7c1e9b97ed1c2a"
format_stamp: "Formatted at 2026-06-19 02:25:13 on dist-test-slave-xkrt"
server_key: "b402cb07926fab0f0545c7e68bb776dd"
server_key_iv: "0230a547570133df6014d45f09081774"
server_key_version: "encryptionkey@0"
I20260619 02:25:13.499763 7874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:13.520855 7874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:13.524458 7874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:13.524684 7874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:13.532992 7874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36889
I20260619 02:25:13.533052 7951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36889 every 8 connection(s)
I20260619 02:25:13.534044 7874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:25:13.536819 7952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:13.538089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 7874
I20260619 02:25:13.538218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:25:13.538532 392 external_mini_cluster.cc:1428] Setting key 9e28e12db84581252f6fedcca19d5cf7
I20260619 02:25:13.543325 7952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a: Bootstrap starting.
I20260619 02:25:13.545979 7952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:13.546907 7952 log.cc:824] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:25:13 dist-test-slave-xkrt krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 02:25:13.549207 7952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a: No bootstrap required, opened a new log
I20260619 02:25:13.553267 7952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3b5ade642d4009bc7c1e9b97ed1c2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } }
I20260619 02:25:13.553572 7952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:13.553622 7952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3b5ade642d4009bc7c1e9b97ed1c2a, State: Initialized, Role: FOLLOWER
I20260619 02:25:13.553962 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:13.539901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.0.98.62:36889)
0619 02:25:13.540276 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:13.540291 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:13.540333 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:25:13.541057 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:13.541083 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:13.541421 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:13.541701 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:13.542560 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:13.543626 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:13.544743 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:13.545202 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:13.545348 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:13.548152 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:13.548158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:13.548173 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:13.548201 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:13.550719 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:13.551159 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:13.551165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:13.551171 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:13.551304 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:13.551564 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:13.551567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:13.551569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:13.552065 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:13.552260 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:13.552635 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 02:25:13.552941 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":117,"threads_started":1}
I20260619 02:25:13.554075 7952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3b5ade642d4009bc7c1e9b97ed1c2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } }
I20260619 02:25:13.554224 7952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:25:13.554292 7952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:25:13.554394 7952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:13.555425 7952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3b5ade642d4009bc7c1e9b97ed1c2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } }
I20260619 02:25:13.555763 7952 leader_election.cc:302] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c3b5ade642d4009bc7c1e9b97ed1c2a; no voters:
I20260619 02:25:13.556108 7952 leader_election.cc:288] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:25:13.556188 7957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:13.556500 7957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [term 1 LEADER]: Becoming Leader. State: Replica: 5c3b5ade642d4009bc7c1e9b97ed1c2a, State: Running, Role: LEADER
I20260619 02:25:13.556854 7957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3b5ade642d4009bc7c1e9b97ed1c2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } }
I20260619 02:25:13.557389 7952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:13.558512 7959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c3b5ade642d4009bc7c1e9b97ed1c2a. Latest consensus state: current_term: 1 leader_uuid: "5c3b5ade642d4009bc7c1e9b97ed1c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3b5ade642d4009bc7c1e9b97ed1c2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } } }
I20260619 02:25:13.558631 7959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:13.558832 7958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c3b5ade642d4009bc7c1e9b97ed1c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3b5ade642d4009bc7c1e9b97ed1c2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } } }
I20260619 02:25:13.559024 7958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:13.559947 7966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:13.562472 7966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:13.567276 7966 catalog_manager.cc:1347] Generated new cluster ID: c3b1902bf06a492f8db2508eacc49652
I20260619 02:25:13.567334 7966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:13.591142 7966 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:25:13.592083 7966 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:13.600006 7966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a: Generated new TSK 0
I20260619 02:25:13.600817 7966 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 02:25:13.629047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:13.738288 7978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:13.738550 7978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:13.738593 7978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:13.742606 7978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:13.742660 7978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:13.742699 7978 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:25:13.742766 7978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:25:13.747843 7978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:13.748827 7978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:13.749912 7978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:13.759869 7983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:13.759836 7984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:13.759879 7986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:13.760280 7978 server_base.cc:1034] running on GCE node
I20260619 02:25:13.760700 7978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:13.761353 7978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:13.762540 7978 hybrid_clock.cc:648] HybridClock initialized: now 1781835913762547 us; error 48 us; skew 500 ppm
Jun 19 02:25:13 dist-test-slave-xkrt krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835913, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:13.765285 7978 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 02:25:13.766242 7978 webserver.cc:460] Webserver started at http://127.0.98.1:36029/ using document root <none> and password file <none>
I20260619 02:25:13.766808 7978 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:13.766909 7978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:13.767122 7978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:13.769095 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "c95ae6aba17f40efab8223fe639054d2"
format_stamp: "Formatted at 2026-06-19 02:25:13 on dist-test-slave-xkrt"
server_key: "6c3b68149288963734a75a2d0f0d95a2"
server_key_iv: "908b34711ddce84621e912d1a4a76ec3"
server_key_version: "encryptionkey@0"
I20260619 02:25:13.769723 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c95ae6aba17f40efab8223fe639054d2"
format_stamp: "Formatted at 2026-06-19 02:25:13 on dist-test-slave-xkrt"
server_key: "6c3b68149288963734a75a2d0f0d95a2"
server_key_iv: "908b34711ddce84621e912d1a4a76ec3"
server_key_version: "encryptionkey@0"
I20260619 02:25:13.773237 7978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:25:13.775805 7993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:13.777287 7978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:25:13.777420 7978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "c95ae6aba17f40efab8223fe639054d2"
format_stamp: "Formatted at 2026-06-19 02:25:13 on dist-test-slave-xkrt"
server_key: "6c3b68149288963734a75a2d0f0d95a2"
server_key_iv: "908b34711ddce84621e912d1a4a76ec3"
server_key_version: "encryptionkey@0"
I20260619 02:25:13.777587 7978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:13.796579 7978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:13.800124 7978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:13.800349 7978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:13.801921 7978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:13.801975 7978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:13.802040 7978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:13.802075 7978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:25:13 dist-test-slave-xkrt krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835913, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 02:25:13.813786 7978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41619
I20260619 02:25:13.813905 8108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41619 every 8 connection(s)
I20260619 02:25:13.814877 7978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:25:13.815392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 7978
I20260619 02:25:13.815529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:25:13.815861 392 external_mini_cluster.cc:1428] Setting key 4611423eb8a2bc1d1e8d70072527bf88
I20260619 02:25:13.818203 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:13.802893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39679 (local address 127.0.98.62:36889)
0619 02:25:13.803061 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:13.803065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:13.804205 (+ 1140us) server_negotiation.cc:407] Connection header received
0619 02:25:13.805590 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:13.805604 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:13.805676 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:13.805745 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:13.807695 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:13.808703 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:13.810733 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:13.811103 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:13.811232 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:13.814258 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:13.814261 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:13.814263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:13.814289 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:13.815978 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:13.816575 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:13.816578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:13.816579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:13.816682 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:13.817009 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:13.817013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:13.817014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:13.817231 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:13.817331 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:13.817966 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 02:25:13.818079 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:25:13.819317 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:13.803461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36889 (local address 127.0.98.1:39679)
0619 02:25:13.803963 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:13.804100 (+ 137us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:13.805271 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:13.806242 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:13.806254 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:13.806701 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:13.807459 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:13.807485 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:13.808868 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:13.808872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:13.810596 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:13.810607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:13.811271 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:13.811275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:13.811475 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:13.812154 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:13.812184 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:13.814070 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:13.816070 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:13.816075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:13.816087 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:13.816439 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:13.816784 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:13.816786 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:13.816787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:13.816917 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:13.817347 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:13.817355 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:13.817664 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 02:25:13.817890 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 02:25:13.818246 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":131,"threads_started":1}
I20260619 02:25:13.820559 8109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36889
I20260619 02:25:13.820693 8109 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:13.821230 8109 heartbeater.cc:511] Master 127.0.98.62:36889 requested a full tablet report, sending...
I20260619 02:25:13.822968 7905 ts_manager.cc:194] Registered new tserver with Master: c95ae6aba17f40efab8223fe639054d2 (127.0.98.1:41619)
I20260619 02:25:13.824136 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39679
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260619 02:25:13.832553 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:13.822880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36889 (local address 127.0.98.1:52723)
0619 02:25:13.824817 (+ 1937us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:13.824830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:13.824956 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:13.825244 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:13.825246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:13.825460 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:13.825596 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:13.825603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:13.826689 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:13.826692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:13.827432 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:13.827442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:13.827907 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:13.827910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:13.828012 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:13.828609 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:13.828621 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:13.828995 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:13.830903 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:13.830908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:13.830912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:13.831208 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:13.831534 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:13.831538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:13.831540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:13.831647 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:13.832117 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:13.832120 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:13.832200 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 02:25:13.832285 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 02:25:13.832407 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1850,"thread_start_us":111,"threads_started":1}
I20260619 02:25:13.832844 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:13.822941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52723 (local address 127.0.98.62:36889)
0619 02:25:13.823051 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:13.823056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:13.824906 (+ 1850us) server_negotiation.cc:407] Connection header received
0619 02:25:13.825070 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:13.825073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:13.825122 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:13.825214 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:13.825721 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:13.826530 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:13.827547 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:13.827765 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:13.827852 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:13.829173 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:13.829180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:13.829186 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:13.829219 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:13.830805 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:13.831355 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:13.831357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:13.831359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:13.831436 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:13.831751 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:13.831754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:13.831755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:13.831987 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:13.832103 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:13.832420 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 02:25:13.832566 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 02:25:13.847954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:13.957331 8118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:13.957613 8118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:13.957661 8118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:13.961777 8118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:13.961831 8118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:13.961869 8118 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:25:13.961900 8118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:25:13.966984 8118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:13.967868 8118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:13.968761 8118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:13.977893 8126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:13.977847 8124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:13.977936 8118 server_base.cc:1034] running on GCE node
W20260619 02:25:13.977847 8123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:13.978456 8118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:13.979075 8118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:13.980252 8118 hybrid_clock.cc:648] HybridClock initialized: now 1781835913980234 us; error 30 us; skew 500 ppm
Jun 19 02:25:13 dist-test-slave-xkrt krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835913, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:13.983211 8118 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 02:25:13.984005 8118 webserver.cc:460] Webserver started at http://127.0.98.2:34633/ using document root <none> and password file <none>
I20260619 02:25:13.984500 8118 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:13.984557 8118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:13.984709 8118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:13.986681 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "c7b1b629727549da960db634af041b0f"
format_stamp: "Formatted at 2026-06-19 02:25:13 on dist-test-slave-xkrt"
server_key: "1d081bf90c68425fa2e7fd4ef1f4489a"
server_key_iv: "091cc5a7dabdab4693a4e6fb0105e4d0"
server_key_version: "encryptionkey@0"
I20260619 02:25:13.987169 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c7b1b629727549da960db634af041b0f"
format_stamp: "Formatted at 2026-06-19 02:25:13 on dist-test-slave-xkrt"
server_key: "1d081bf90c68425fa2e7fd4ef1f4489a"
server_key_iv: "091cc5a7dabdab4693a4e6fb0105e4d0"
server_key_version: "encryptionkey@0"
I20260619 02:25:13.990561 8118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:25:13.992987 8133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:13.994220 8118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:13.994349 8118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "c7b1b629727549da960db634af041b0f"
format_stamp: "Formatted at 2026-06-19 02:25:13 on dist-test-slave-xkrt"
server_key: "1d081bf90c68425fa2e7fd4ef1f4489a"
server_key_iv: "091cc5a7dabdab4693a4e6fb0105e4d0"
server_key_version: "encryptionkey@0"
I20260619 02:25:13.994468 8118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:14.004885 8118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:14.007578 8118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:14.007761 8118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:14.009053 8118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:14.009124 8118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:14.009191 8118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:14.009243 8118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:25:14 dist-test-slave-xkrt krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835913, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 02:25:14.019825 8118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39853
I20260619 02:25:14.019905 8248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39853 every 8 connection(s)
I20260619 02:25:14.020761 8118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:25:14.023727 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.009959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43557 (local address 127.0.98.62:36889)
0619 02:25:14.010082 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.010086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.011071 (+ 985us) server_negotiation.cc:407] Connection header received
0619 02:25:14.012126 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.012130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.012193 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.012256 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:14.013686 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.014599 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.015860 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.016271 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.016400 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.020013 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:14.020020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:14.020025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:14.020053 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:14.021795 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:14.022269 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:14.022274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:14.022276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:14.022370 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:14.022691 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:14.022695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:14.022697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:14.022880 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:14.022990 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.023440 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.023561 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:25:14.023767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 8118
I20260619 02:25:14.023934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:25:14.024214 392 external_mini_cluster.cc:1428] Setting key 372231d32642687588cdd764dbde62b0
I20260619 02:25:14.024538 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.010410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36889 (local address 127.0.98.2:43557)
0619 02:25:14.010919 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:14.010959 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:14.011881 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:14.012627 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:14.012635 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:14.013004 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:14.013494 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.013513 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.014798 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.014801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.015747 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.015756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.016452 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.016457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.016670 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.017994 (+ 1324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:14.018020 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:14.019802 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:14.021906 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:14.021911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:14.021924 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:14.022166 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:14.022467 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:14.022470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:14.022471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:14.022599 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:14.022985 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:14.022991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:14.023212 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 02:25:14.023400 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 02:25:14.023683 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":122,"threads_started":1}
I20260619 02:25:14.025645 8249 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36889
I20260619 02:25:14.025805 8249 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:14.026255 8249 heartbeater.cc:511] Master 127.0.98.62:36889 requested a full tablet report, sending...
I20260619 02:25:14.027226 7905 ts_manager.cc:194] Registered new tserver with Master: c7b1b629727549da960db634af041b0f (127.0.98.2:39853)
I20260619 02:25:14.028054 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43557
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:25:14.035529 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.028634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60747 (local address 127.0.98.62:36889)
0619 02:25:14.028742 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.028746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.028977 (+ 231us) server_negotiation.cc:407] Connection header received
0619 02:25:14.029109 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.029112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.029150 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.029239 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:14.029681 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.030261 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.031046 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.031271 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.031375 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.032419 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:14.032423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:14.032426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:14.032448 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:14.033954 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:14.034375 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:14.034380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:14.034382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:14.034458 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:14.034732 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:14.034738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:14.034740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:14.034937 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:14.035043 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.035285 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.035381 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:25:14.035543 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.028568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36889 (local address 127.0.98.2:60747)
0619 02:25:14.028896 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:14.028907 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:14.029016 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:14.029258 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:14.029261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:14.029455 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:14.029586 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.029595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.030380 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.030383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.030935 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.030942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.031378 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.031381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.031467 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.031964 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:14.031978 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:14.032275 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:14.034061 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:14.034065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:14.034067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:14.034273 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:14.034556 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:14.034559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:14.034561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:14.034641 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:14.035033 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:14.035035 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:14.035128 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 02:25:14.035235 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 02:25:14.035375 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 02:25:14.050659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:14.158869 8258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:14.159142 8258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:14.159212 8258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:14.163149 8258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:14.163234 8258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:14.163317 8258 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:25:14.163403 8258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:25:14.168583 8258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:14.169608 8258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:14.170586 8258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:25:14.180224 8258 server_base.cc:1034] running on GCE node
W20260619 02:25:14.180234 8263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:14.180375 8266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:14.180234 8264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:14.180840 8258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:14.181577 8258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:14.182770 8258 hybrid_clock.cc:648] HybridClock initialized: now 1781835914182757 us; error 26 us; skew 500 ppm
Jun 19 02:25:14 dist-test-slave-xkrt krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835914, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:14.185513 8258 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 02:25:14.186403 8258 webserver.cc:460] Webserver started at http://127.0.98.3:39433/ using document root <none> and password file <none>
I20260619 02:25:14.186936 8258 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:14.186997 8258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:14.187201 8258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:14.189124 8258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "c3d823b3044a4267869ee5409b0957ee"
format_stamp: "Formatted at 2026-06-19 02:25:14 on dist-test-slave-xkrt"
server_key: "72393b6741d17786fb58ea17f59f12d7"
server_key_iv: "77980b318a5495f9ab782a41c474224b"
server_key_version: "encryptionkey@0"
I20260619 02:25:14.189728 8258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c3d823b3044a4267869ee5409b0957ee"
format_stamp: "Formatted at 2026-06-19 02:25:14 on dist-test-slave-xkrt"
server_key: "72393b6741d17786fb58ea17f59f12d7"
server_key_iv: "77980b318a5495f9ab782a41c474224b"
server_key_version: "encryptionkey@0"
I20260619 02:25:14.193122 8258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:25:14.195544 8273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:14.196789 8258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:14.196924 8258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "c3d823b3044a4267869ee5409b0957ee"
format_stamp: "Formatted at 2026-06-19 02:25:14 on dist-test-slave-xkrt"
server_key: "72393b6741d17786fb58ea17f59f12d7"
server_key_iv: "77980b318a5495f9ab782a41c474224b"
server_key_version: "encryptionkey@0"
I20260619 02:25:14.197072 8258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:14.224798 8258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:14.227808 8258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:14.228011 8258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:14.229460 8258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:14.229534 8258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:14.229631 8258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:14.229687 8258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:25:14 dist-test-slave-xkrt krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835914, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 02:25:14.240496 8258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36285
I20260619 02:25:14.240756 8388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36285 every 8 connection(s)
I20260619 02:25:14.241513 8258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:25:14.243180 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.230211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59893 (local address 127.0.98.62:36889)
0619 02:25:14.230360 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.230364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.231218 (+ 854us) server_negotiation.cc:407] Connection header received
0619 02:25:14.232170 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.232174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.232235 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.232340 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:14.233864 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.234397 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.235758 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.236024 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.236146 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.239611 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:14.239614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:14.239616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:14.239639 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:14.241195 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:14.241769 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:14.241772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:14.241773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:14.241852 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:14.242253 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:14.242256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:14.242257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:14.242493 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:14.242570 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.242972 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.243064 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:25:14.243919 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.230612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36889 (local address 127.0.98.3:59893)
0619 02:25:14.231049 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:14.231096 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:14.231927 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:14.232651 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:14.232677 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:14.233144 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:14.233718 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.233733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.234522 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.234525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.235640 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.235649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.236177 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.236181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.236419 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.236984 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:14.237026 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:14.239450 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:14.241302 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:14.241307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:14.241320 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:14.241680 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:14.241972 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:14.241975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:14.241979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:14.242142 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:14.242574 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:14.242579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:14.242775 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 02:25:14.242920 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 02:25:14.243140 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":125,"threads_started":1}
I20260619 02:25:14.245191 8389 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36889
I20260619 02:25:14.245398 8389 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:14.245914 8389 heartbeater.cc:511] Master 127.0.98.62:36889 requested a full tablet report, sending...
I20260619 02:25:14.246321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 8258
I20260619 02:25:14.246433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:25:14.246798 392 external_mini_cluster.cc:1428] Setting key 5813114d6bfb5dacd172c03ddfb538fd
I20260619 02:25:14.246974 7905 ts_manager.cc:194] Registered new tserver with Master: c3d823b3044a4267869ee5409b0957ee (127.0.98.3:36285)
I20260619 02:25:14.247721 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59893
I20260619 02:25:14.248747 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:25:14.255586 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.247405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36889 (local address 127.0.98.3:60411)
0619 02:25:14.247655 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:14.247668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:14.247788 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:14.248080 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:14.248083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:14.248265 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:14.248392 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.248399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.249422 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.249425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.250131 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.250139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.250791 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.250795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.250885 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.251488 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:14.251510 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:14.252009 (+ 499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:14.253810 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:14.253817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:14.253820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:14.254098 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:14.254421 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:14.254425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:14.254428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:14.254528 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:14.254917 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:14.254923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:14.255034 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 02:25:14.255149 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 02:25:14.255313 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
I20260619 02:25:14.255981 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.247381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60411 (local address 127.0.98.62:36889)
0619 02:25:14.247491 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.247496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.247726 (+ 230us) server_negotiation.cc:407] Connection header received
0619 02:25:14.247930 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.247933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.247974 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.248066 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:14.248520 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.249263 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.250257 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.250641 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.250756 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.252147 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:14.252150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:14.252152 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:14.252171 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:14.253693 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:14.254239 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:14.254242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:14.254243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:14.254321 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:14.254629 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:14.254631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:14.254633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:14.254818 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:14.254900 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.255793 (+ 893us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.255885 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 02:25:14 dist-test-slave-xkrt krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835914, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:25:14 dist-test-slave-xkrt krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:14 dist-test-slave-xkrt krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:25:14.273497 7955 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:14.267578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48364 (local address 127.0.98.62:36889)
0619 02:25:14.267720 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.267726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.267746 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:25:14.267777 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.267779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.267811 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.267904 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:14.268569 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.269079 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.269977 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.270189 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.270295 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.273347 (+ 3052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48364: BlockingRecv error: recv got EOF from 127.0.0.1:48364 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 02:25:14 dist-test-slave-xkrt krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835914, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 02:25:14.283474 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.274874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.0.98.62:36889)
0619 02:25:14.275008 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.275013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.275155 (+ 142us) server_negotiation.cc:407] Connection header received
0619 02:25:14.275293 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.275296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.275339 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.275419 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:14.275989 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.276480 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.277237 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.277452 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.277582 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.280299 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:14.280303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:14.280306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:14.280330 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:14.281879 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:14.282282 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:14.282286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:14.282288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:14.282370 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:14.282646 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:14.282649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:14.282651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:14.282832 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:14.282967 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.283191 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.283316 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:25:14.286595 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:25:14.289628 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:25:14.303916 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.298814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.0.98.2:39853)
0619 02:25:14.299397 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.299409 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.299430 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:25:14.299501 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.299506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.299678 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.299821 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:14.299996 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.301090 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.302620 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.303234 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.303376 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.303603 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.303667 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.303753 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":103,"threads_started":1}
I20260619 02:25:14.303968 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.298111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39853 (local address 127.0.0.1:44652)
0619 02:25:14.298453 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:14.298467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:14.298554 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:14.299794 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:14.299796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:14.299815 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:14.299890 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.299895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.301193 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.301199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.302498 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.302507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.303380 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.303384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.303489 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.303526 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 02:25:14.303621 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 02:25:14.303776 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":28,"spinlock_wait_cycles":384,"thread_start_us":81,"threads_started":1}
I20260619 02:25:14.304129 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.298522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41619 (local address 127.0.0.1:52218)
0619 02:25:14.299024 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:14.299037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:14.299131 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:14.299771 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:14.299775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:14.299793 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:14.299895 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.299900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.301214 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.301217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.302553 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.302577 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.303783 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.303786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.303876 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.303894 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 02:25:14.303952 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 02:25:14.304010 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":116,"spinlock_wait_cycles":3328,"thread_start_us":82,"threads_started":1}
I20260619 02:25:14.304258 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.297878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.0.98.3:36285)
0619 02:25:14.298540 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.298551 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.298574 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:25:14.298653 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.298659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.298883 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.299081 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:14.299394 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.300553 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.302487 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.303331 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.303438 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.303604 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.304028 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.304109 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":130,"threads_started":1}
I20260619 02:25:14.304366 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.297750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36285 (local address 127.0.0.1:59902)
0619 02:25:14.298061 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:14.298104 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:14.298220 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:14.299069 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:14.299075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:14.299097 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:14.299257 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.299265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.300679 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.300682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.302395 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.302404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.303665 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.303668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.303765 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.303790 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:25:14.303857 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 02:25:14.303927 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":195,"thread_start_us":88,"threads_started":1}
I20260619 02:25:14.304615 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.298665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.0.98.1:41619)
0619 02:25:14.298963 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.298978 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.299223 (+ 245us) server_negotiation.cc:407] Connection header received
0619 02:25:14.299301 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.299306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.299488 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.299626 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:14.300041 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.301090 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.302673 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.303489 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.303583 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.303723 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.304348 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.304459 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":95,"threads_started":1}
I20260619 02:25:14.307037 8043 tablet_service.cc:1467] Processing CreateTablet for tablet c4989efc658c4c3e966fc8ec4e8c67e1 (DEFAULT_TABLE table=test-table [id=ac6b03c18dd1475ea5fc3209a08c66c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:14.307134 8323 tablet_service.cc:1467] Processing CreateTablet for tablet c4989efc658c4c3e966fc8ec4e8c67e1 (DEFAULT_TABLE table=test-table [id=ac6b03c18dd1475ea5fc3209a08c66c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:14.307132 8183 tablet_service.cc:1467] Processing CreateTablet for tablet c4989efc658c4c3e966fc8ec4e8c67e1 (DEFAULT_TABLE table=test-table [id=ac6b03c18dd1475ea5fc3209a08c66c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:14.307996 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4989efc658c4c3e966fc8ec4e8c67e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:14.308065 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4989efc658c4c3e966fc8ec4e8c67e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:14.308059 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4989efc658c4c3e966fc8ec4e8c67e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:14.314884 8417 tablet_bootstrap.cc:492] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2: Bootstrap starting.
I20260619 02:25:14.315946 8419 tablet_bootstrap.cc:492] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f: Bootstrap starting.
I20260619 02:25:14.317093 8417 tablet_bootstrap.cc:654] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:14.317658 8419 tablet_bootstrap.cc:654] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:14.317888 8418 tablet_bootstrap.cc:492] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee: Bootstrap starting.
I20260619 02:25:14.317950 8417 log.cc:824] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:14.318506 8419 log.cc:824] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:14.319897 8418 tablet_bootstrap.cc:654] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:14.319906 8417 tablet_bootstrap.cc:492] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2: No bootstrap required, opened a new log
I20260619 02:25:14.320101 8417 ts_tablet_manager.cc:1397] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:25:14.320434 8419 tablet_bootstrap.cc:492] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f: No bootstrap required, opened a new log
I20260619 02:25:14.320611 8419 ts_tablet_manager.cc:1397] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:25:14.320794 8418 log.cc:824] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:14.322916 8418 tablet_bootstrap.cc:492] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee: No bootstrap required, opened a new log
I20260619 02:25:14.323089 8418 ts_tablet_manager.cc:1397] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 02:25:14.323582 8417 raft_consensus.cc:348] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260619 02:25:14.323819 8417 raft_consensus.cc:374] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:14.323890 8417 raft_consensus.cc:729] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95ae6aba17f40efab8223fe639054d2, State: Initialized, Role: FOLLOWER
I20260619 02:25:14.324334 8417 consensus_queue.cc:260] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260619 02:25:14.324452 8419 raft_consensus.cc:348] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260619 02:25:14.324738 8419 raft_consensus.cc:374] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:14.324826 8419 raft_consensus.cc:729] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7b1b629727549da960db634af041b0f, State: Initialized, Role: FOLLOWER
I20260619 02:25:14.324935 8109 heartbeater.cc:503] Master 127.0.98.62:36889 was elected leader, sending a full tablet report...
I20260619 02:25:14.325374 8417 ts_tablet_manager.cc:1428] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:25:14.325352 8419 consensus_queue.cc:260] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260619 02:25:14.326124 8249 heartbeater.cc:503] Master 127.0.98.62:36889 was elected leader, sending a full tablet report...
I20260619 02:25:14.326426 8419 ts_tablet_manager.cc:1428] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:25:14.326973 8418 raft_consensus.cc:348] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260619 02:25:14.327240 8418 raft_consensus.cc:374] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:14.327324 8418 raft_consensus.cc:729] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d823b3044a4267869ee5409b0957ee, State: Initialized, Role: FOLLOWER
I20260619 02:25:14.327750 8418 consensus_queue.cc:260] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260619 02:25:14.328676 8418 ts_tablet_manager.cc:1428] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:25:14.328981 8389 heartbeater.cc:503] Master 127.0.98.62:36889 was elected leader, sending a full tablet report...
W20260619 02:25:14.492653 8390 tablet.cc:2369] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:25:14.522035 8250 tablet.cc:2369] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:25:14.566948 8110 tablet.cc:2369] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:25:14.592542 8425 raft_consensus.cc:482] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:25:14.592809 8425 raft_consensus.cc:504] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260619 02:25:14.594053 8425 leader_election.cc:288] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c95ae6aba17f40efab8223fe639054d2 (127.0.98.1:41619), c7b1b629727549da960db634af041b0f (127.0.98.2:39853)
I20260619 02:25:14.598095 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.594427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53695 (local address 127.0.98.1:41619)
0619 02:25:14.594580 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.594584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.594607 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:25:14.594814 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.594818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.594874 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.594964 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:14.595254 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.595813 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.596933 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.597392 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.597524 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.597617 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.597898 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.597973 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:25:14.598107 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.594344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41619 (local address 127.0.98.3:53695)
0619 02:25:14.594546 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:14.594563 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:14.594666 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:14.594987 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:14.594990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:14.595007 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:14.595134 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.595139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.595932 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.595939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.596798 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.596806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.597524 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.597528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.597690 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.597741 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 02:25:14.597859 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 02:25:14.597940 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260619 02:25:14.598412 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.594585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54841 (local address 127.0.98.2:39853)
0619 02:25:14.594730 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.594735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.594890 (+ 155us) server_negotiation.cc:407] Connection header received
0619 02:25:14.595046 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.595049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.595099 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.595184 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:14.595792 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.596270 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.597261 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.597775 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.597908 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.597966 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.598215 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.598307 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:25:14.598754 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.594390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39853 (local address 127.0.98.3:54841)
0619 02:25:14.594784 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:14.594795 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:14.594942 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:14.595193 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:14.595196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:14.595210 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:14.595670 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.595677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.596391 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.596395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.597130 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.597138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.597931 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.597935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.598039 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.598077 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 02:25:14.598561 (+ 484us) client_negotiation.cc:240] Negotiation successful
0619 02:25:14.598637 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":116,"threads_started":1}
I20260619 02:25:14.598753 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4989efc658c4c3e966fc8ec4e8c67e1" candidate_uuid: "c3d823b3044a4267869ee5409b0957ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95ae6aba17f40efab8223fe639054d2" is_pre_election: true
I20260619 02:25:14.599071 8063 raft_consensus.cc:2454] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d823b3044a4267869ee5409b0957ee in term 0.
I20260619 02:25:14.599380 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4989efc658c4c3e966fc8ec4e8c67e1" candidate_uuid: "c3d823b3044a4267869ee5409b0957ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7b1b629727549da960db634af041b0f" is_pre_election: true
I20260619 02:25:14.599609 8275 leader_election.cc:302] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d823b3044a4267869ee5409b0957ee, c95ae6aba17f40efab8223fe639054d2; no voters:
I20260619 02:25:14.599673 8203 raft_consensus.cc:2454] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3d823b3044a4267869ee5409b0957ee in term 0.
I20260619 02:25:14.599912 8425 raft_consensus.cc:2781] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:25:14.600030 8425 raft_consensus.cc:482] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:25:14.600096 8425 raft_consensus.cc:3037] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:14.601223 8425 raft_consensus.cc:504] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260619 02:25:14.601680 8425 leader_election.cc:288] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [CANDIDATE]: Term 1 election: Requested vote from peers c95ae6aba17f40efab8223fe639054d2 (127.0.98.1:41619), c7b1b629727549da960db634af041b0f (127.0.98.2:39853)
I20260619 02:25:14.601943 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4989efc658c4c3e966fc8ec4e8c67e1" candidate_uuid: "c3d823b3044a4267869ee5409b0957ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95ae6aba17f40efab8223fe639054d2"
I20260619 02:25:14.601977 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4989efc658c4c3e966fc8ec4e8c67e1" candidate_uuid: "c3d823b3044a4267869ee5409b0957ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7b1b629727549da960db634af041b0f"
I20260619 02:25:14.602090 8203 raft_consensus.cc:3037] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:14.602084 8063 raft_consensus.cc:3037] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:14.603102 8063 raft_consensus.cc:2454] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d823b3044a4267869ee5409b0957ee in term 1.
I20260619 02:25:14.603229 8203 raft_consensus.cc:2454] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3d823b3044a4267869ee5409b0957ee in term 1.
I20260619 02:25:14.603499 8275 leader_election.cc:302] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d823b3044a4267869ee5409b0957ee, c95ae6aba17f40efab8223fe639054d2; no voters:
I20260619 02:25:14.603688 8425 raft_consensus.cc:2781] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:14.603931 8425 raft_consensus.cc:686] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [term 1 LEADER]: Becoming Leader. State: Replica: c3d823b3044a4267869ee5409b0957ee, State: Running, Role: LEADER
I20260619 02:25:14.604243 8425 consensus_queue.cc:237] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260619 02:25:14.607262 7906 catalog_manager.cc:5495] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee reported cstate change: term changed from 0 to 1, leader changed from <none> to c3d823b3044a4267869ee5409b0957ee (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c3d823b3044a4267869ee5409b0957ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } health_report { overall_health: UNKNOWN } } }
I20260619 02:25:14.667138 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.663502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.0.98.3:36285)
0619 02:25:14.663654 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.663659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.663675 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:25:14.663807 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.663810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.663872 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.663988 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:25:14.664248 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.664798 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.665668 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.665895 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.666005 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.666299 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:25:14.666736 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:25:14.666865 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.666930 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.666993 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:25:14.672700 8203 raft_consensus.cc:1261] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Refusing update from remote peer c3d823b3044a4267869ee5409b0957ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:14.672701 8063 raft_consensus.cc:1261] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Refusing update from remote peer c3d823b3044a4267869ee5409b0957ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:14.673451 8425 consensus_queue.cc:1035] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:14.673702 8427 consensus_queue.cc:1035] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:14.682873 8430 mvcc.cc:205] Tried to move back new op lower bound from 7298399906492342272 to 7298399906221674496. Current Snapshot: MvccSnapshot[applied={T|T < 7298399906492342272 or (T in {7298399906492342272})}]
I20260619 02:25:14.688802 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:14.688977 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:14.691552 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a: Sending DeleteTablet for 3 replicas of tablet c4989efc658c4c3e966fc8ec4e8c67e1
I20260619 02:25:14.692346 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet c4989efc658c4c3e966fc8ec4e8c67e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59902
I20260619 02:25:14.692378 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet c4989efc658c4c3e966fc8ec4e8c67e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44652
I20260619 02:25:14.692517 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet c4989efc658c4c3e966fc8ec4e8c67e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52218
I20260619 02:25:14.692823 8439 tablet_replica.cc:330] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f: stopping tablet replica
I20260619 02:25:14.692970 8440 tablet_replica.cc:330] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee: stopping tablet replica
I20260619 02:25:14.693184 8439 raft_consensus.cc:2229] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:14.693289 8440 raft_consensus.cc:2229] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:25:14.693655 8439 raft_consensus.cc:2258] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:14.693673 8440 raft_consensus.cc:2258] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:14.693749 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:25:14.694211 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:25:14.694233 8441 tablet_replica.cc:330] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2: stopping tablet replica
I20260619 02:25:14.694497 8441 raft_consensus.cc:2229] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:14.694748 8441 raft_consensus.cc:2258] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:14.695370 8440 ts_tablet_manager.cc:1905] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:14.696089 8441 ts_tablet_manager.cc:1905] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:14.698449 8440 ts_tablet_manager.cc:1918] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:14.698621 8440 log.cc:1196] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/c4989efc658c4c3e966fc8ec4e8c67e1
I20260619 02:25:14.698776 8441 ts_tablet_manager.cc:1918] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:14.698940 8441 log.cc:1196] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/wal/wals/c4989efc658c4c3e966fc8ec4e8c67e1
I20260619 02:25:14.699054 8440 ts_tablet_manager.cc:1939] T c4989efc658c4c3e966fc8ec4e8c67e1 P c3d823b3044a4267869ee5409b0957ee: Deleting consensus metadata
I20260619 02:25:14.699317 8441 ts_tablet_manager.cc:1939] T c4989efc658c4c3e966fc8ec4e8c67e1 P c95ae6aba17f40efab8223fe639054d2: Deleting consensus metadata
I20260619 02:25:14.699945 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 57420a39399c4b2b933be237c8da62f0 (DEFAULT_TABLE table=test-table [id=3076793da4364357ac5965b03c005aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:14.700199 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57420a39399c4b2b933be237c8da62f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:14.701037 8439 ts_tablet_manager.cc:1905] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:14.702472 8418 tablet_bootstrap.cc:492] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee: Bootstrap starting.
I20260619 02:25:14.702708 7891 catalog_manager.cc:4841] TS c3d823b3044a4267869ee5409b0957ee (127.0.98.3:36285): tablet c4989efc658c4c3e966fc8ec4e8c67e1 (table test-table [id=ac6b03c18dd1475ea5fc3209a08c66c4]) successfully deleted
I20260619 02:25:14.703258 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 57420a39399c4b2b933be237c8da62f0 (DEFAULT_TABLE table=test-table [id=3076793da4364357ac5965b03c005aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:14.703426 8418 tablet_bootstrap.cc:654] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:14.703533 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57420a39399c4b2b933be237c8da62f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:14.703895 8439 ts_tablet_manager.cc:1918] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:14.703976 8439 log.cc:1196] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/c4989efc658c4c3e966fc8ec4e8c67e1
I20260619 02:25:14.704279 8439 ts_tablet_manager.cc:1939] T c4989efc658c4c3e966fc8ec4e8c67e1 P c7b1b629727549da960db634af041b0f: Deleting consensus metadata
I20260619 02:25:14.704691 8418 tablet_bootstrap.cc:492] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee: No bootstrap required, opened a new log
I20260619 02:25:14.704779 8418 ts_tablet_manager.cc:1397] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:14.705185 8418 raft_consensus.cc:348] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.705281 8418 raft_consensus.cc:374] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:14.705310 8418 raft_consensus.cc:729] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d823b3044a4267869ee5409b0957ee, State: Initialized, Role: FOLLOWER
I20260619 02:25:14.705425 8418 consensus_queue.cc:260] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.705679 8418 ts_tablet_manager.cc:1428] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:25:14.705974 8419 tablet_bootstrap.cc:492] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f: Bootstrap starting.
I20260619 02:25:14.706816 8419 tablet_bootstrap.cc:654] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:14.708173 7893 catalog_manager.cc:4841] TS c7b1b629727549da960db634af041b0f (127.0.98.2:39853): tablet c4989efc658c4c3e966fc8ec4e8c67e1 (table test-table [id=ac6b03c18dd1475ea5fc3209a08c66c4]) successfully deleted
I20260619 02:25:14.708458 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 57420a39399c4b2b933be237c8da62f0 (DEFAULT_TABLE table=test-table [id=3076793da4364357ac5965b03c005aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:14.708724 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57420a39399c4b2b933be237c8da62f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:14.709692 7891 catalog_manager.cc:4841] TS c95ae6aba17f40efab8223fe639054d2 (127.0.98.1:41619): tablet c4989efc658c4c3e966fc8ec4e8c67e1 (table test-table [id=ac6b03c18dd1475ea5fc3209a08c66c4]) successfully deleted
I20260619 02:25:14.710229 8419 tablet_bootstrap.cc:492] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f: No bootstrap required, opened a new log
I20260619 02:25:14.710330 8419 ts_tablet_manager.cc:1397] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 02:25:14.710808 8419 raft_consensus.cc:348] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.710925 8419 raft_consensus.cc:374] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:14.710958 8417 tablet_bootstrap.cc:492] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2: Bootstrap starting.
I20260619 02:25:14.710968 8419 raft_consensus.cc:729] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7b1b629727549da960db634af041b0f, State: Initialized, Role: FOLLOWER
I20260619 02:25:14.711109 8419 consensus_queue.cc:260] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.711445 8419 ts_tablet_manager.cc:1428] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:25:14.711864 8417 tablet_bootstrap.cc:654] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:14.713063 8417 tablet_bootstrap.cc:492] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2: No bootstrap required, opened a new log
I20260619 02:25:14.713140 8417 ts_tablet_manager.cc:1397] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:14.713860 8417 raft_consensus.cc:348] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.713968 8417 raft_consensus.cc:374] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:14.713999 8417 raft_consensus.cc:729] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95ae6aba17f40efab8223fe639054d2, State: Initialized, Role: FOLLOWER
I20260619 02:25:14.714100 8417 consensus_queue.cc:260] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.714764 8417 ts_tablet_manager.cc:1428] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:25:14.787472 8423 raft_consensus.cc:482] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:25:14.787634 8423 raft_consensus.cc:504] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.788861 8423 leader_election.cc:288] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7b1b629727549da960db634af041b0f (127.0.98.2:39853), c3d823b3044a4267869ee5409b0957ee (127.0.98.3:36285)
I20260619 02:25:14.793855 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.789493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47833 (local address 127.0.98.2:39853)
0619 02:25:14.789649 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.789653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.789715 (+ 62us) server_negotiation.cc:407] Connection header received
0619 02:25:14.789888 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.789891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.789942 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.790010 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:14.790460 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.791178 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.792581 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.793191 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.793297 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.793376 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.793628 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.793722 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:25:14.793880 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.789134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39853 (local address 127.0.98.1:47833)
0619 02:25:14.789617 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:14.789630 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:14.789746 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:14.790107 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:14.790116 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:14.790130 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:14.790303 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.790312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.791330 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.791333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.792447 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.792455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.793335 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.793339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.793451 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.793477 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 02:25:14.793609 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 02:25:14.793681 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":62,"thread_start_us":65,"threads_started":1}
I20260619 02:25:14.793923 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.789206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50761 (local address 127.0.98.3:36285)
0619 02:25:14.789341 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.789347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.789617 (+ 270us) server_negotiation.cc:407] Connection header received
0619 02:25:14.789855 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.789860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.789910 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.789987 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:14.790329 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.791142 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.792725 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.793312 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.793412 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.793468 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.793730 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.793812 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:25:14.794329 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.789134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36285 (local address 127.0.98.1:50761)
0619 02:25:14.789502 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:14.789518 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:14.789713 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:14.790010 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:14.790013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:14.790030 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:14.790213 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.790219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.791266 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.791270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.792585 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.792599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.793445 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.793449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.793589 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.793615 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 02:25:14.794103 (+ 488us) client_negotiation.cc:240] Negotiation successful
0619 02:25:14.794193 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":62,"thread_start_us":97,"threads_started":1}
I20260619 02:25:14.794528 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57420a39399c4b2b933be237c8da62f0" candidate_uuid: "c95ae6aba17f40efab8223fe639054d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7b1b629727549da960db634af041b0f" is_pre_election: true
I20260619 02:25:14.794692 8203 raft_consensus.cc:2454] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c95ae6aba17f40efab8223fe639054d2 in term 0.
I20260619 02:25:14.794931 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57420a39399c4b2b933be237c8da62f0" candidate_uuid: "c95ae6aba17f40efab8223fe639054d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d823b3044a4267869ee5409b0957ee" is_pre_election: true
I20260619 02:25:14.795171 8343 raft_consensus.cc:2454] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c95ae6aba17f40efab8223fe639054d2 in term 0.
I20260619 02:25:14.795313 7997 leader_election.cc:302] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7b1b629727549da960db634af041b0f, c95ae6aba17f40efab8223fe639054d2; no voters:
I20260619 02:25:14.795567 8423 raft_consensus.cc:2781] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:25:14.795629 8423 raft_consensus.cc:482] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:25:14.795671 8423 raft_consensus.cc:3037] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:14.796587 8423 raft_consensus.cc:504] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.796904 8423 leader_election.cc:288] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [CANDIDATE]: Term 1 election: Requested vote from peers c7b1b629727549da960db634af041b0f (127.0.98.2:39853), c3d823b3044a4267869ee5409b0957ee (127.0.98.3:36285)
I20260619 02:25:14.797248 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57420a39399c4b2b933be237c8da62f0" candidate_uuid: "c95ae6aba17f40efab8223fe639054d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7b1b629727549da960db634af041b0f"
I20260619 02:25:14.797412 8203 raft_consensus.cc:3037] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:14.797664 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57420a39399c4b2b933be237c8da62f0" candidate_uuid: "c95ae6aba17f40efab8223fe639054d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d823b3044a4267869ee5409b0957ee"
I20260619 02:25:14.797775 8343 raft_consensus.cc:3037] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:14.798605 8203 raft_consensus.cc:2454] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95ae6aba17f40efab8223fe639054d2 in term 1.
I20260619 02:25:14.798925 8343 raft_consensus.cc:2454] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95ae6aba17f40efab8223fe639054d2 in term 1.
I20260619 02:25:14.798923 7997 leader_election.cc:302] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7b1b629727549da960db634af041b0f, c95ae6aba17f40efab8223fe639054d2; no voters:
I20260619 02:25:14.799218 8423 raft_consensus.cc:2781] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:14.799448 8423 raft_consensus.cc:686] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [term 1 LEADER]: Becoming Leader. State: Replica: c95ae6aba17f40efab8223fe639054d2, State: Running, Role: LEADER
I20260619 02:25:14.799697 8423 consensus_queue.cc:237] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.802170 7906 catalog_manager.cc:5495] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c95ae6aba17f40efab8223fe639054d2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c95ae6aba17f40efab8223fe639054d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:25:14 dist-test-slave-xkrt krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835913, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 02:25:14.848419 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.839438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.0.98.62:36889)
0619 02:25:14.839730 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.839739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.839753 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:25:14.839891 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.839894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.839941 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.840036 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:14.840670 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.841212 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.842033 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.842245 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.842352 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.844723 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:14.844725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:14.844727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:14.844749 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:14.846576 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:14.847087 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:14.847090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:14.847091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:14.847160 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:14.847454 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:14.847457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:14.847458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:14.847668 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:14.847764 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.848070 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.848209 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":214,"thread_start_us":109,"threads_started":1}
I20260619 02:25:14.848438 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.839294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36889 (local address 127.0.0.1:34574)
0619 02:25:14.839636 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:14.839653 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:14.839770 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:14.840051 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:14.840054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:14.840458 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:14.840553 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.840559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.841330 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.841333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.841926 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.841934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.842350 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.842353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.842486 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.843081 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:14.843093 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:14.844583 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:14.846678 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:14.846686 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:14.846692 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:14.846988 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:14.847239 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:14.847242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:14.847244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:14.847334 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:14.847771 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:14.847777 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:14.847892 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 02:25:14.848003 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:25:14.848151 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":36,"thread_start_us":127,"threads_started":1}
I20260619 02:25:14.852528 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34574:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:25:14.853297 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:25:14.858413 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 488aeabf48d449a5bbff1c0e773650d3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a52954ec8344485784da52a66fd649d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:25:14.858412 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 488aeabf48d449a5bbff1c0e773650d3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a52954ec8344485784da52a66fd649d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:25:14.858479 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 488aeabf48d449a5bbff1c0e773650d3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a52954ec8344485784da52a66fd649d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:25:14.858783 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488aeabf48d449a5bbff1c0e773650d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:14.858783 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488aeabf48d449a5bbff1c0e773650d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:14.858870 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488aeabf48d449a5bbff1c0e773650d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:14.861366 8419 tablet_bootstrap.cc:492] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f: Bootstrap starting.
I20260619 02:25:14.861457 8418 tablet_bootstrap.cc:492] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee: Bootstrap starting.
I20260619 02:25:14.861457 8417 tablet_bootstrap.cc:492] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2: Bootstrap starting.
I20260619 02:25:14.862267 8418 tablet_bootstrap.cc:654] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:14.862267 8417 tablet_bootstrap.cc:654] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:14.863494 8418 tablet_bootstrap.cc:492] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee: No bootstrap required, opened a new log
I20260619 02:25:14.863587 8418 ts_tablet_manager.cc:1397] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:14.863684 8419 tablet_bootstrap.cc:654] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:14.863902 8417 tablet_bootstrap.cc:492] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2: No bootstrap required, opened a new log
I20260619 02:25:14.863993 8417 ts_tablet_manager.cc:1397] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 02:25:14.863971 8418 raft_consensus.cc:348] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.864087 8418 raft_consensus.cc:374] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:14.864135 8418 raft_consensus.cc:729] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d823b3044a4267869ee5409b0957ee, State: Initialized, Role: FOLLOWER
I20260619 02:25:14.864254 8418 consensus_queue.cc:260] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.864434 8417 raft_consensus.cc:348] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.864522 8418 ts_tablet_manager.cc:1428] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:25:14.864548 8417 raft_consensus.cc:374] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:14.864580 8417 raft_consensus.cc:729] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95ae6aba17f40efab8223fe639054d2, State: Initialized, Role: FOLLOWER
I20260619 02:25:14.864692 8417 consensus_queue.cc:260] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.864730 8427 tablet_replica.cc:439] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } }
I20260619 02:25:14.864837 8427 tablet_replica.cc:442] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:25:14.864946 8419 tablet_bootstrap.cc:492] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f: No bootstrap required, opened a new log
I20260619 02:25:14.864962 8417 ts_tablet_manager.cc:1428] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:25:14.865027 8419 ts_tablet_manager.cc:1397] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 02:25:14.865168 8423 tablet_replica.cc:439] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } }
I20260619 02:25:14.865278 8423 tablet_replica.cc:442] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:25:14.865499 8419 raft_consensus.cc:348] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.865649 8419 raft_consensus.cc:374] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:14.865696 8419 raft_consensus.cc:729] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7b1b629727549da960db634af041b0f, State: Initialized, Role: FOLLOWER
I20260619 02:25:14.865862 8419 consensus_queue.cc:260] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.866151 8419 ts_tablet_manager.cc:1428] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:25:14.866364 8424 tablet_replica.cc:439] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } }
I20260619 02:25:14.866482 8424 tablet_replica.cc:442] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:25:14.992460 8424 raft_consensus.cc:482] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:25:14.992641 8424 raft_consensus.cc:504] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:14.994114 8424 leader_election.cc:288] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c95ae6aba17f40efab8223fe639054d2 (127.0.98.1:41619), c3d823b3044a4267869ee5409b0957ee (127.0.98.3:36285)
I20260619 02:25:14.999459 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.994308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41619 (local address 127.0.98.2:39253)
0619 02:25:14.994685 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:14.994704 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:14.994842 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:14.995124 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:14.995127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:14.995148 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:14.995297 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.995305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.996516 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.996519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.998066 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.998080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.998995 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.998998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.999111 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.999133 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:25:14.999231 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 02:25:14.999295 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":10,"thread_start_us":123,"threads_started":1}
I20260619 02:25:14.999435 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.994447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39253 (local address 127.0.98.1:41619)
0619 02:25:14.994585 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.994589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.994913 (+ 324us) server_negotiation.cc:407] Connection header received
0619 02:25:14.994976 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.994979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.995030 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.995089 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:14.995437 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.996415 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.998180 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.998882 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.998985 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.999051 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.999245 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.999314 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:25:15.000000 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.994810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37571 (local address 127.0.98.3:36285)
0619 02:25:14.996405 (+ 1595us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:14.996408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:14.996421 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:25:14.996484 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:14.996487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:14.996539 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:14.996645 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:14.996872 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.997652 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.999096 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.999470 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.999588 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.999644 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:14.999810 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 02:25:14.999881 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1488}
I20260619 02:25:15.000228 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488aeabf48d449a5bbff1c0e773650d3" candidate_uuid: "c7b1b629727549da960db634af041b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95ae6aba17f40efab8223fe639054d2" is_pre_election: true
I20260619 02:25:15.000409 8063 raft_consensus.cc:2454] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7b1b629727549da960db634af041b0f in term 0.
I20260619 02:25:15.000746 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:14.994722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36285 (local address 127.0.98.2:37571)
0619 02:25:14.995255 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:14.995266 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:14.995330 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:14.996650 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:14.996652 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:14.996669 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:14.996760 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.996764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.997787 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.997791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.998912 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:14.998921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:14.999588 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:14.999591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:14.999685 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:14.999707 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:25:15.000591 (+ 884us) client_negotiation.cc:240] Negotiation successful
0619 02:25:15.000646 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":80,"threads_started":1}
I20260619 02:25:15.000799 8135 leader_election.cc:302] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7b1b629727549da960db634af041b0f, c95ae6aba17f40efab8223fe639054d2; no voters:
I20260619 02:25:15.001068 8424 raft_consensus.cc:2781] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:25:15.001144 8424 raft_consensus.cc:482] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:25:15.001199 8424 raft_consensus.cc:3037] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:15.001376 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488aeabf48d449a5bbff1c0e773650d3" candidate_uuid: "c7b1b629727549da960db634af041b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d823b3044a4267869ee5409b0957ee" is_pre_election: true
I20260619 02:25:15.001529 8343 raft_consensus.cc:2454] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7b1b629727549da960db634af041b0f in term 0.
I20260619 02:25:15.002471 8424 raft_consensus.cc:504] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.002746 8424 leader_election.cc:288] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [CANDIDATE]: Term 1 election: Requested vote from peers c95ae6aba17f40efab8223fe639054d2 (127.0.98.1:41619), c3d823b3044a4267869ee5409b0957ee (127.0.98.3:36285)
I20260619 02:25:15.003093 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488aeabf48d449a5bbff1c0e773650d3" candidate_uuid: "c7b1b629727549da960db634af041b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95ae6aba17f40efab8223fe639054d2"
I20260619 02:25:15.003171 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488aeabf48d449a5bbff1c0e773650d3" candidate_uuid: "c7b1b629727549da960db634af041b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d823b3044a4267869ee5409b0957ee"
I20260619 02:25:15.003259 8063 raft_consensus.cc:3037] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:15.003307 8343 raft_consensus.cc:3037] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:15.004142 8063 raft_consensus.cc:2454] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7b1b629727549da960db634af041b0f in term 1.
I20260619 02:25:15.004227 8343 raft_consensus.cc:2454] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7b1b629727549da960db634af041b0f in term 1.
I20260619 02:25:15.004429 8135 leader_election.cc:302] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7b1b629727549da960db634af041b0f, c95ae6aba17f40efab8223fe639054d2; no voters:
I20260619 02:25:15.004618 8424 raft_consensus.cc:2781] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:15.004814 8424 raft_consensus.cc:686] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [term 1 LEADER]: Becoming Leader. State: Replica: c7b1b629727549da960db634af041b0f, State: Running, Role: LEADER
I20260619 02:25:15.005084 8424 consensus_queue.cc:237] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.006280 8461 tablet_replica.cc:439] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f: TxnStatusTablet state changed. Reason: New leader c7b1b629727549da960db634af041b0f. Latest consensus state: current_term: 1 leader_uuid: "c7b1b629727549da960db634af041b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } }
I20260619 02:25:15.006389 8461 tablet_replica.cc:442] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:25:15.006639 8463 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:25:15.006784 8463 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:25:15.007462 7905 catalog_manager.cc:5495] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f reported cstate change: term changed from 0 to 1, leader changed from <none> to c7b1b629727549da960db634af041b0f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c7b1b629727549da960db634af041b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } health_report { overall_health: UNKNOWN } } }
I20260619 02:25:15.029336 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:15.019565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40834 (local address 127.0.98.2:39853)
0619 02:25:15.025733 (+ 6168us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:15.025738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:15.025755 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:25:15.025825 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:15.025828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:15.025892 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:15.026009 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:25:15.026257 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.026859 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.027860 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.028103 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.028217 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:15.028521 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:25:15.028969 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:25:15.029093 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:15.029163 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 02:25:15.029212 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6073}
I20260619 02:25:15.029363 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:15.019449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39853 (local address 127.0.0.1:40834)
0619 02:25:15.019593 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:15.019608 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:15.019752 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:15.026023 (+ 6271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:15.026026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:15.026034 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:25:15.026144 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:15.026149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.026971 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.026975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:15.027654 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:15.027662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.028218 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.028222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:15.028331 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:15.028374 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:25:15.029070 (+ 696us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:25:15.029084 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 02:25:15.029153 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 02:25:15.029237 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 02:25:15.032070 8063 raft_consensus.cc:1261] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Refusing update from remote peer c7b1b629727549da960db634af041b0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:15.032258 8343 raft_consensus.cc:1261] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Refusing update from remote peer c7b1b629727549da960db634af041b0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:15.032521 8461 consensus_queue.cc:1035] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:15.032766 8424 consensus_queue.cc:1035] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:15.034340 8423 tablet_replica.cc:439] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2: TxnStatusTablet state changed. Reason: New leader c7b1b629727549da960db634af041b0f. Latest consensus state: current_term: 1 leader_uuid: "c7b1b629727549da960db634af041b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } }
I20260619 02:25:15.034471 8423 tablet_replica.cc:442] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:25:15.034626 8427 tablet_replica.cc:439] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee: TxnStatusTablet state changed. Reason: New leader c7b1b629727549da960db634af041b0f. Latest consensus state: current_term: 1 leader_uuid: "c7b1b629727549da960db634af041b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } }
I20260619 02:25:15.034744 8427 tablet_replica.cc:442] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:25:15.036070 8424 tablet_replica.cc:439] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7b1b629727549da960db634af041b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } }
I20260619 02:25:15.036172 8424 tablet_replica.cc:442] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:25:15.036548 8431 mvcc.cc:205] Tried to move back new op lower bound from 7298399907966398464 to 7298399907863461888. Current Snapshot: MvccSnapshot[applied={T|T < 7298399907966398464 or (T in {7298399907966398464})}]
I20260619 02:25:15.037421 8427 tablet_replica.cc:439] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7b1b629727549da960db634af041b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } }
I20260619 02:25:15.037525 8427 tablet_replica.cc:442] T 488aeabf48d449a5bbff1c0e773650d3 P c3d823b3044a4267869ee5409b0957ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:25:15.038484 8424 tablet_replica.cc:439] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7b1b629727549da960db634af041b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } }
I20260619 02:25:15.038589 8424 tablet_replica.cc:442] T 488aeabf48d449a5bbff1c0e773650d3 P c7b1b629727549da960db634af041b0f: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:25:15.039000 8432 mvcc.cc:205] Tried to move back new op lower bound from 7298399907966398464 to 7298399907863461888. Current Snapshot: MvccSnapshot[applied={T|T < 7298399907966398464}]
I20260619 02:25:15.039140 8423 tablet_replica.cc:439] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7b1b629727549da960db634af041b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } }
I20260619 02:25:15.039260 8423 tablet_replica.cc:442] T 488aeabf48d449a5bbff1c0e773650d3 P c95ae6aba17f40efab8223fe639054d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:25:15.044744 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:15.041139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60146 (local address 127.0.98.1:41619)
0619 02:25:15.041279 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:15.041283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:15.041297 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:25:15.041364 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:15.041367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:15.041424 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:15.041559 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:25:15.041789 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.042411 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.043356 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.043616 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.043732 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:15.043991 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:25:15.044380 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:25:15.044499 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:15.044565 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 02:25:15.044614 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:25:15.054965 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:15.051217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39853 (local address 127.0.98.1:43661)
0619 02:25:15.051587 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:15.051600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:15.051678 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:15.051923 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:15.051925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:15.051932 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:25:15.052031 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:15.052035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.053010 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.053012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:15.053723 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:15.053732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.054279 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.054283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:15.054359 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:15.054399 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:25:15.054702 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:25:15.054709 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:25:15.054775 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 02:25:15.054837 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":190,"threads_started":1}
I20260619 02:25:15.054965 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:15.051332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43661 (local address 127.0.98.2:39853)
0619 02:25:15.051458 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:15.051462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:15.051667 (+ 205us) server_negotiation.cc:407] Connection header received
0619 02:25:15.051747 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:15.051750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:15.051799 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:15.051882 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:25:15.052139 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.052900 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.053834 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.054166 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.054267 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:15.054519 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:25:15.054594 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:25:15.054668 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:15.054803 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 02:25:15.054851 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:25:15.061084 8203 raft_consensus.cc:1261] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Refusing update from remote peer c95ae6aba17f40efab8223fe639054d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:15.061553 8447 consensus_queue.cc:1035] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:15.061659 8343 raft_consensus.cc:1261] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Refusing update from remote peer c95ae6aba17f40efab8223fe639054d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:15.062155 8447 consensus_queue.cc:1035] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:15.083810 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:15.079576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45563 (local address 127.0.98.1:41619)
0619 02:25:15.079700 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:15.079705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:15.079949 (+ 244us) server_negotiation.cc:407] Connection header received
0619 02:25:15.080101 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:15.080104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:15.080152 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:15.080238 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:25:15.080502 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.081267 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.082621 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.082833 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.082936 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:15.083216 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:25:15.083418 (+ 202us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:25:15.083491 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:15.083623 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 02:25:15.083676 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:25:15.084400 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:15.079472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41619 (local address 127.0.98.2:45563)
0619 02:25:15.079843 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:15.079877 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:15.079995 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:15.080254 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:15.080257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:15.080264 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:25:15.080388 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:15.080394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.081403 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.081406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:15.082507 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:15.082518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.082957 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.082960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:15.083053 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:15.083102 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:25:15.083507 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:25:15.083513 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:25:15.084139 (+ 626us) client_negotiation.cc:240] Negotiation successful
0619 02:25:15.084216 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":93,"threads_started":1}
I20260619 02:25:15.110503 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:15.110671 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:15.112885 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a: Sending DeleteTablet for 3 replicas of tablet 57420a39399c4b2b933be237c8da62f0
I20260619 02:25:15.113451 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 57420a39399c4b2b933be237c8da62f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59902
I20260619 02:25:15.113639 8440 tablet_replica.cc:330] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee: stopping tablet replica
I20260619 02:25:15.113673 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 57420a39399c4b2b933be237c8da62f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44652
I20260619 02:25:15.113758 8440 raft_consensus.cc:2229] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:15.113837 8439 tablet_replica.cc:330] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f: stopping tablet replica
I20260619 02:25:15.113859 8440 raft_consensus.cc:2258] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:15.113950 8439 raft_consensus.cc:2229] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:15.114059 8439 raft_consensus.cc:2258] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:15.114781 8440 ts_tablet_manager.cc:1905] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:15.117472 8440 ts_tablet_manager.cc:1918] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:25:15.117636 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 57420a39399c4b2b933be237c8da62f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52218
I20260619 02:25:15.117779 8441 tablet_replica.cc:330] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2: stopping tablet replica
I20260619 02:25:15.117869 8441 raft_consensus.cc:2229] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:25:15.118129 8441 raft_consensus.cc:2258] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:15.118273 8440 log.cc:1196] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/57420a39399c4b2b933be237c8da62f0
I20260619 02:25:15.118583 8440 ts_tablet_manager.cc:1939] T 57420a39399c4b2b933be237c8da62f0 P c3d823b3044a4267869ee5409b0957ee: Deleting consensus metadata
I20260619 02:25:15.118868 8439 ts_tablet_manager.cc:1905] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:15.118893 8441 ts_tablet_manager.cc:1905] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:15.119212 7891 catalog_manager.cc:4841] TS c3d823b3044a4267869ee5409b0957ee (127.0.98.3:36285): tablet 57420a39399c4b2b933be237c8da62f0 (table test-table [id=3076793da4364357ac5965b03c005aa4]) successfully deleted
I20260619 02:25:15.121528 8439 ts_tablet_manager.cc:1918] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:25:15.121577 8441 ts_tablet_manager.cc:1918] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:25:15.121613 8439 log.cc:1196] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/57420a39399c4b2b933be237c8da62f0
I20260619 02:25:15.121641 8441 log.cc:1196] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/wal/wals/57420a39399c4b2b933be237c8da62f0
I20260619 02:25:15.121874 8439 ts_tablet_manager.cc:1939] T 57420a39399c4b2b933be237c8da62f0 P c7b1b629727549da960db634af041b0f: Deleting consensus metadata
I20260619 02:25:15.121892 8441 ts_tablet_manager.cc:1939] T 57420a39399c4b2b933be237c8da62f0 P c95ae6aba17f40efab8223fe639054d2: Deleting consensus metadata
I20260619 02:25:15.122469 7891 catalog_manager.cc:4841] TS c95ae6aba17f40efab8223fe639054d2 (127.0.98.1:41619): tablet 57420a39399c4b2b933be237c8da62f0 (table test-table [id=3076793da4364357ac5965b03c005aa4]) successfully deleted
I20260619 02:25:15.122498 7893 catalog_manager.cc:4841] TS c7b1b629727549da960db634af041b0f (127.0.98.2:39853): tablet 57420a39399c4b2b933be237c8da62f0 (table test-table [id=3076793da4364357ac5965b03c005aa4]) successfully deleted
Jun 19 02:25:15 dist-test-slave-xkrt krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:25:15 dist-test-slave-xkrt krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:15 dist-test-slave-xkrt krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:25:15.155351 8455 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:15.144549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.0.98.62:36889)
0619 02:25:15.144685 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:15.144690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:15.144972 (+ 282us) server_negotiation.cc:407] Connection header received
0619 02:25:15.145107 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:15.145112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:15.145172 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:15.145278 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:15.146192 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.147048 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.148206 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.148566 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.148671 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:15.155214 (+ 6543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34578: BlockingRecv error: recv got EOF from 127.0.0.1:34578 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 02:25:15 dist-test-slave-xkrt krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 02:25:15.167536 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:15.157060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.0.98.62:36889)
0619 02:25:15.157167 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:15.157171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:15.157314 (+ 143us) server_negotiation.cc:407] Connection header received
0619 02:25:15.157492 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:15.157495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:15.157572 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:15.157667 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:15.158448 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.159273 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.160417 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.160738 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.160834 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:15.163858 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:15.163861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:15.163864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:15.163898 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:15.165790 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:15.166271 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:15.166275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:15.166277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:15.166360 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:15.166652 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:15.166655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:15.166657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:15.166870 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:15.167010 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:15.167246 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 02:25:15.167365 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:25:15.170512 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:25:15.170991 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:25:15.176292 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 8abafb5f5b0341ef865b1b633e4dc504 (DEFAULT_TABLE table=test-table [id=4b2f4ee7c36c4c558e533bf3a5549c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:15.176355 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 8abafb5f5b0341ef865b1b633e4dc504 (DEFAULT_TABLE table=test-table [id=4b2f4ee7c36c4c558e533bf3a5549c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:15.176582 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8abafb5f5b0341ef865b1b633e4dc504. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:15.176635 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8abafb5f5b0341ef865b1b633e4dc504. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:15.179111 8418 tablet_bootstrap.cc:492] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee: Bootstrap starting.
I20260619 02:25:15.179301 8419 tablet_bootstrap.cc:492] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f: Bootstrap starting.
I20260619 02:25:15.179913 8418 tablet_bootstrap.cc:654] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:15.180179 8419 tablet_bootstrap.cc:654] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:15.181041 8418 tablet_bootstrap.cc:492] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee: No bootstrap required, opened a new log
I20260619 02:25:15.181149 8418 ts_tablet_manager.cc:1397] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:25:15.181123 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 8abafb5f5b0341ef865b1b633e4dc504 (DEFAULT_TABLE table=test-table [id=4b2f4ee7c36c4c558e533bf3a5549c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:15.181412 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8abafb5f5b0341ef865b1b633e4dc504. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:15.181627 8418 raft_consensus.cc:348] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.181757 8418 raft_consensus.cc:374] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:15.181797 8418 raft_consensus.cc:729] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d823b3044a4267869ee5409b0957ee, State: Initialized, Role: FOLLOWER
I20260619 02:25:15.181936 8418 consensus_queue.cc:260] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.182197 8418 ts_tablet_manager.cc:1428] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:25:15.183878 8417 tablet_bootstrap.cc:492] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2: Bootstrap starting.
I20260619 02:25:15.184782 8417 tablet_bootstrap.cc:654] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:15.184863 8419 tablet_bootstrap.cc:492] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f: No bootstrap required, opened a new log
I20260619 02:25:15.184959 8419 ts_tablet_manager.cc:1397] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260619 02:25:15.185353 8419 raft_consensus.cc:348] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.185482 8419 raft_consensus.cc:374] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:15.185913 8417 tablet_bootstrap.cc:492] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2: No bootstrap required, opened a new log
I20260619 02:25:15.185997 8417 ts_tablet_manager.cc:1397] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:15.185524 8419 raft_consensus.cc:729] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7b1b629727549da960db634af041b0f, State: Initialized, Role: FOLLOWER
I20260619 02:25:15.186316 8419 consensus_queue.cc:260] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.186414 8417 raft_consensus.cc:348] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.186532 8417 raft_consensus.cc:374] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:15.186560 8417 raft_consensus.cc:729] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95ae6aba17f40efab8223fe639054d2, State: Initialized, Role: FOLLOWER
I20260619 02:25:15.186609 8419 ts_tablet_manager.cc:1428] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 02:25:15.186666 8417 consensus_queue.cc:260] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.186892 8417 ts_tablet_manager.cc:1428] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:25:15.355934 8447 raft_consensus.cc:482] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:25:15.356127 8447 raft_consensus.cc:504] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.356637 8447 leader_election.cc:288] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7b1b629727549da960db634af041b0f (127.0.98.2:39853), c3d823b3044a4267869ee5409b0957ee (127.0.98.3:36285)
I20260619 02:25:15.357112 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abafb5f5b0341ef865b1b633e4dc504" candidate_uuid: "c95ae6aba17f40efab8223fe639054d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7b1b629727549da960db634af041b0f" is_pre_election: true
I20260619 02:25:15.357129 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abafb5f5b0341ef865b1b633e4dc504" candidate_uuid: "c95ae6aba17f40efab8223fe639054d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d823b3044a4267869ee5409b0957ee" is_pre_election: true
I20260619 02:25:15.357285 8203 raft_consensus.cc:2454] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c95ae6aba17f40efab8223fe639054d2 in term 0.
I20260619 02:25:15.357306 8343 raft_consensus.cc:2454] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c95ae6aba17f40efab8223fe639054d2 in term 0.
I20260619 02:25:15.357595 7997 leader_election.cc:302] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7b1b629727549da960db634af041b0f, c95ae6aba17f40efab8223fe639054d2; no voters:
I20260619 02:25:15.357812 8447 raft_consensus.cc:2781] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:25:15.357884 8447 raft_consensus.cc:482] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:25:15.357937 8447 raft_consensus.cc:3037] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:15.359025 8447 raft_consensus.cc:504] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.359390 8447 leader_election.cc:288] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [CANDIDATE]: Term 1 election: Requested vote from peers c7b1b629727549da960db634af041b0f (127.0.98.2:39853), c3d823b3044a4267869ee5409b0957ee (127.0.98.3:36285)
I20260619 02:25:15.359805 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abafb5f5b0341ef865b1b633e4dc504" candidate_uuid: "c95ae6aba17f40efab8223fe639054d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7b1b629727549da960db634af041b0f"
I20260619 02:25:15.359836 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abafb5f5b0341ef865b1b633e4dc504" candidate_uuid: "c95ae6aba17f40efab8223fe639054d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d823b3044a4267869ee5409b0957ee"
I20260619 02:25:15.359944 8203 raft_consensus.cc:3037] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:15.359968 8343 raft_consensus.cc:3037] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:15.361044 8203 raft_consensus.cc:2454] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95ae6aba17f40efab8223fe639054d2 in term 1.
I20260619 02:25:15.361209 8343 raft_consensus.cc:2454] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95ae6aba17f40efab8223fe639054d2 in term 1.
I20260619 02:25:15.361387 7997 leader_election.cc:302] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7b1b629727549da960db634af041b0f, c95ae6aba17f40efab8223fe639054d2; no voters:
I20260619 02:25:15.361613 8447 raft_consensus.cc:2781] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:15.361742 8447 raft_consensus.cc:686] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [term 1 LEADER]: Becoming Leader. State: Replica: c95ae6aba17f40efab8223fe639054d2, State: Running, Role: LEADER
I20260619 02:25:15.361886 8447 consensus_queue.cc:237] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.363626 7906 catalog_manager.cc:5495] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c95ae6aba17f40efab8223fe639054d2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c95ae6aba17f40efab8223fe639054d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } health_report { overall_health: UNKNOWN } } }
I20260619 02:25:15.382236 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:15.378844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.0.98.1:41619)
0619 02:25:15.378991 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:15.378998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:15.379014 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:25:15.379171 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:15.379181 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:15.379224 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:15.379306 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:25:15.379548 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.380047 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.381027 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.381291 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.381400 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:15.381724 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:25:15.381810 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:25:15.381877 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:15.381989 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 02:25:15.382032 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:25:15.384167 8343 raft_consensus.cc:1261] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Refusing update from remote peer c95ae6aba17f40efab8223fe639054d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:15.384389 8203 raft_consensus.cc:1261] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Refusing update from remote peer c95ae6aba17f40efab8223fe639054d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:15.384617 8423 consensus_queue.cc:1035] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:15.385149 8423 consensus_queue.cc:1035] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:15.391606 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:15.391768 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:15.393671 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a: Sending DeleteTablet for 3 replicas of tablet 8abafb5f5b0341ef865b1b633e4dc504
I20260619 02:25:15.394150 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 8abafb5f5b0341ef865b1b633e4dc504 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52218
I20260619 02:25:15.394291 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 8abafb5f5b0341ef865b1b633e4dc504 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44652
I20260619 02:25:15.394325 8441 tablet_replica.cc:330] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2: stopping tablet replica
I20260619 02:25:15.394405 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 8abafb5f5b0341ef865b1b633e4dc504 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59902
I20260619 02:25:15.394445 8439 tablet_replica.cc:330] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f: stopping tablet replica
I20260619 02:25:15.394483 8441 raft_consensus.cc:2229] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:25:15.394553 8440 tablet_replica.cc:330] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee: stopping tablet replica
I20260619 02:25:15.394553 8439 raft_consensus.cc:2229] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:15.394650 8440 raft_consensus.cc:2229] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:15.394665 8439 raft_consensus.cc:2258] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:15.394675 8441 raft_consensus.cc:2258] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:15.394765 8440 raft_consensus.cc:2258] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:15.395291 8441 ts_tablet_manager.cc:1905] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:15.395454 8439 ts_tablet_manager.cc:1905] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:15.395820 8440 ts_tablet_manager.cc:1905] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:15.397408 8441 ts_tablet_manager.cc:1918] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:15.397505 8441 log.cc:1196] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-0/wal/wals/8abafb5f5b0341ef865b1b633e4dc504
I20260619 02:25:15.397822 8441 ts_tablet_manager.cc:1939] T 8abafb5f5b0341ef865b1b633e4dc504 P c95ae6aba17f40efab8223fe639054d2: Deleting consensus metadata
I20260619 02:25:15.397884 8439 ts_tablet_manager.cc:1918] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:15.397964 8439 log.cc:1196] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/8abafb5f5b0341ef865b1b633e4dc504
I20260619 02:25:15.398190 8440 ts_tablet_manager.cc:1918] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:15.398211 8439 ts_tablet_manager.cc:1939] T 8abafb5f5b0341ef865b1b633e4dc504 P c7b1b629727549da960db634af041b0f: Deleting consensus metadata
I20260619 02:25:15.398262 8440 log.cc:1196] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/8abafb5f5b0341ef865b1b633e4dc504
I20260619 02:25:15.398521 8440 ts_tablet_manager.cc:1939] T 8abafb5f5b0341ef865b1b633e4dc504 P c3d823b3044a4267869ee5409b0957ee: Deleting consensus metadata
I20260619 02:25:15.398900 7891 catalog_manager.cc:4841] TS c95ae6aba17f40efab8223fe639054d2 (127.0.98.1:41619): tablet 8abafb5f5b0341ef865b1b633e4dc504 (table test-table [id=4b2f4ee7c36c4c558e533bf3a5549c74]) successfully deleted
I20260619 02:25:15.398964 7893 catalog_manager.cc:4841] TS c7b1b629727549da960db634af041b0f (127.0.98.2:39853): tablet 8abafb5f5b0341ef865b1b633e4dc504 (table test-table [id=4b2f4ee7c36c4c558e533bf3a5549c74]) successfully deleted
I20260619 02:25:15.399168 7891 catalog_manager.cc:4841] TS c3d823b3044a4267869ee5409b0957ee (127.0.98.3:36285): tablet 8abafb5f5b0341ef865b1b633e4dc504 (table test-table [id=4b2f4ee7c36c4c558e533bf3a5549c74]) successfully deleted
I20260619 02:25:15.399251 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:25:15.399667 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:25:15.403831 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6348ffe8e74d03a85353dd311c20c2 (DEFAULT_TABLE table=test-table [id=c3bdfe2932154dffba642f9a4dd04447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:15.403905 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6348ffe8e74d03a85353dd311c20c2 (DEFAULT_TABLE table=test-table [id=c3bdfe2932154dffba642f9a4dd04447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:15.404088 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6348ffe8e74d03a85353dd311c20c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:15.404162 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6348ffe8e74d03a85353dd311c20c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:15.405922 8417 tablet_bootstrap.cc:492] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2: Bootstrap starting.
I20260619 02:25:15.405978 8419 tablet_bootstrap.cc:492] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f: Bootstrap starting.
I20260619 02:25:15.406057 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6348ffe8e74d03a85353dd311c20c2 (DEFAULT_TABLE table=test-table [id=c3bdfe2932154dffba642f9a4dd04447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:15.406279 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6348ffe8e74d03a85353dd311c20c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:15.406716 8417 tablet_bootstrap.cc:654] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:15.406772 8419 tablet_bootstrap.cc:654] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:15.407734 8417 tablet_bootstrap.cc:492] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2: No bootstrap required, opened a new log
I20260619 02:25:15.407817 8417 ts_tablet_manager.cc:1397] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:25:15.408108 8419 tablet_bootstrap.cc:492] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f: No bootstrap required, opened a new log
I20260619 02:25:15.408174 8419 ts_tablet_manager.cc:1397] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:25:15.408229 8417 raft_consensus.cc:348] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.408322 8417 raft_consensus.cc:374] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:15.408352 8417 raft_consensus.cc:729] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95ae6aba17f40efab8223fe639054d2, State: Initialized, Role: FOLLOWER
I20260619 02:25:15.408458 8417 consensus_queue.cc:260] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.408564 8418 tablet_bootstrap.cc:492] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee: Bootstrap starting.
I20260619 02:25:15.408550 8419 raft_consensus.cc:348] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.408636 8419 raft_consensus.cc:374] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:15.408667 8419 raft_consensus.cc:729] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7b1b629727549da960db634af041b0f, State: Initialized, Role: FOLLOWER
I20260619 02:25:15.408720 8417 ts_tablet_manager.cc:1428] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:25:15.408784 8419 consensus_queue.cc:260] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.409039 8419 ts_tablet_manager.cc:1428] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:25:15.409350 8418 tablet_bootstrap.cc:654] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:15.410468 8418 tablet_bootstrap.cc:492] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee: No bootstrap required, opened a new log
I20260619 02:25:15.410552 8418 ts_tablet_manager.cc:1397] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:25:15.410971 8418 raft_consensus.cc:348] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.411080 8418 raft_consensus.cc:374] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:15.411114 8418 raft_consensus.cc:729] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d823b3044a4267869ee5409b0957ee, State: Initialized, Role: FOLLOWER
I20260619 02:25:15.411235 8418 consensus_queue.cc:260] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.411496 8418 ts_tablet_manager.cc:1428] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:25:15.505430 8468 raft_consensus.cc:482] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:25:15.505628 8468 raft_consensus.cc:504] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.506101 8468 leader_election.cc:288] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c95ae6aba17f40efab8223fe639054d2 (127.0.98.1:41619), c3d823b3044a4267869ee5409b0957ee (127.0.98.3:36285)
I20260619 02:25:15.506553 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6348ffe8e74d03a85353dd311c20c2" candidate_uuid: "c7b1b629727549da960db634af041b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95ae6aba17f40efab8223fe639054d2" is_pre_election: true
I20260619 02:25:15.506582 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6348ffe8e74d03a85353dd311c20c2" candidate_uuid: "c7b1b629727549da960db634af041b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d823b3044a4267869ee5409b0957ee" is_pre_election: true
I20260619 02:25:15.506718 8063 raft_consensus.cc:2454] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7b1b629727549da960db634af041b0f in term 0.
I20260619 02:25:15.506739 8343 raft_consensus.cc:2454] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7b1b629727549da960db634af041b0f in term 0.
I20260619 02:25:15.507067 8135 leader_election.cc:302] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d823b3044a4267869ee5409b0957ee, c7b1b629727549da960db634af041b0f; no voters:
I20260619 02:25:15.507249 8468 raft_consensus.cc:2781] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:25:15.507308 8468 raft_consensus.cc:482] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:25:15.507351 8468 raft_consensus.cc:3037] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:15.508303 8468 raft_consensus.cc:504] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.508611 8468 leader_election.cc:288] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [CANDIDATE]: Term 1 election: Requested vote from peers c95ae6aba17f40efab8223fe639054d2 (127.0.98.1:41619), c3d823b3044a4267869ee5409b0957ee (127.0.98.3:36285)
I20260619 02:25:15.508972 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6348ffe8e74d03a85353dd311c20c2" candidate_uuid: "c7b1b629727549da960db634af041b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95ae6aba17f40efab8223fe639054d2"
I20260619 02:25:15.508987 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6348ffe8e74d03a85353dd311c20c2" candidate_uuid: "c7b1b629727549da960db634af041b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d823b3044a4267869ee5409b0957ee"
I20260619 02:25:15.509096 8063 raft_consensus.cc:3037] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:15.509114 8343 raft_consensus.cc:3037] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:15.510041 8063 raft_consensus.cc:2454] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7b1b629727549da960db634af041b0f in term 1.
I20260619 02:25:15.510089 8343 raft_consensus.cc:2454] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7b1b629727549da960db634af041b0f in term 1.
I20260619 02:25:15.510299 8135 leader_election.cc:302] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7b1b629727549da960db634af041b0f, c95ae6aba17f40efab8223fe639054d2; no voters:
I20260619 02:25:15.510509 8468 raft_consensus.cc:2781] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:15.510598 8468 raft_consensus.cc:686] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [term 1 LEADER]: Becoming Leader. State: Replica: c7b1b629727549da960db634af041b0f, State: Running, Role: LEADER
I20260619 02:25:15.510732 8468 consensus_queue.cc:237] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } }
I20260619 02:25:15.512385 7904 catalog_manager.cc:5495] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f reported cstate change: term changed from 0 to 1, leader changed from <none> to c7b1b629727549da960db634af041b0f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c7b1b629727549da960db634af041b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7b1b629727549da960db634af041b0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } health_report { overall_health: UNKNOWN } } }
I20260619 02:25:15.537599 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:15.534419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.0.98.2:39853)
0619 02:25:15.534526 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:15.534530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:15.534553 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:25:15.534695 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:15.534698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:15.534738 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:15.534798 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:25:15.535031 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.535560 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.536663 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.536876 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.536954 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:15.537193 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:25:15.537253 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:25:15.537304 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:15.537399 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 02:25:15.537440 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:25:15.552610 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:15.543606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39853 (local address 127.0.98.2:37153)
0619 02:25:15.543804 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:15.543837 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:15.543928 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:15.547072 (+ 3144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:15.547075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:15.547081 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:25:15.547180 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:15.547190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.550644 (+ 3454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.550647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:15.551520 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:15.551530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.551972 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.551975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:15.552060 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:15.552093 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:25:15.552350 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:25:15.552357 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:25:15.552421 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:25:15.552504 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":3}
I20260619 02:25:15.552610 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:15.546704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37153 (local address 127.0.98.2:39853)
0619 02:25:15.546844 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:15.546848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:15.546861 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:25:15.546902 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:15.546905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:15.546958 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:15.547055 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:25:15.549607 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.550536 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.551637 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:15.551866 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:15.551972 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:15.552195 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:25:15.552265 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:25:15.552349 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:15.552428 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 02:25:15.552473 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:25:15.559422 8063 raft_consensus.cc:1261] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Refusing update from remote peer c7b1b629727549da960db634af041b0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:15.559576 8343 raft_consensus.cc:1261] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Refusing update from remote peer c7b1b629727549da960db634af041b0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:15.559731 8468 consensus_queue.cc:1035] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95ae6aba17f40efab8223fe639054d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:15.559927 8424 consensus_queue.cc:1035] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d823b3044a4267869ee5409b0957ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:15.595670 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:15.595849 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:15.597963 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c3b5ade642d4009bc7c1e9b97ed1c2a: Sending DeleteTablet for 3 replicas of tablet 0d6348ffe8e74d03a85353dd311c20c2
I20260619 02:25:15.598629 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6348ffe8e74d03a85353dd311c20c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52218
I20260619 02:25:15.598749 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6348ffe8e74d03a85353dd311c20c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59902
I20260619 02:25:15.598827 8441 tablet_replica.cc:330] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2: stopping tablet replica
I20260619 02:25:15.598950 8441 raft_consensus.cc:2229] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:15.599069 8441 raft_consensus.cc:2258] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:15.599140 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6348ffe8e74d03a85353dd311c20c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44652
I20260619 02:25:15.599233 8440 tablet_replica.cc:330] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee: stopping tablet replica
I20260619 02:25:15.599310 8439 tablet_replica.cc:330] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f: stopping tablet replica
I20260619 02:25:15.599351 8440 raft_consensus.cc:2229] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:15.599429 8439 raft_consensus.cc:2229] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:25:15.599459 8440 raft_consensus.cc:2258] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:15.599735 8439 raft_consensus.cc:2258] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:15.600219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 7978
I20260619 02:25:15.600312 8441 ts_tablet_manager.cc:1905] T 0d6348ffe8e74d03a85353dd311c20c2 P c95ae6aba17f40efab8223fe639054d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:15.601091 8440 ts_tablet_manager.cc:1905] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:15.603574 8440 ts_tablet_manager.cc:1918] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:25:15.603600 8439 ts_tablet_manager.cc:1905] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:15.603647 8440 log.cc:1196] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/0d6348ffe8e74d03a85353dd311c20c2
I20260619 02:25:15.603900 8440 ts_tablet_manager.cc:1939] T 0d6348ffe8e74d03a85353dd311c20c2 P c3d823b3044a4267869ee5409b0957ee: Deleting consensus metadata
I20260619 02:25:15.606680 8439 ts_tablet_manager.cc:1918] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:25:15.606768 8439 log.cc:1196] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/0d6348ffe8e74d03a85353dd311c20c2
I20260619 02:25:15.607043 8439 ts_tablet_manager.cc:1939] T 0d6348ffe8e74d03a85353dd311c20c2 P c7b1b629727549da960db634af041b0f: Deleting consensus metadata
I20260619 02:25:15.607085 7891 catalog_manager.cc:4841] TS c3d823b3044a4267869ee5409b0957ee (127.0.98.3:36285): tablet 0d6348ffe8e74d03a85353dd311c20c2 (table test-table [id=c3bdfe2932154dffba642f9a4dd04447]) successfully deleted
I20260619 02:25:15.607753 7893 catalog_manager.cc:4841] TS c7b1b629727549da960db634af041b0f (127.0.98.2:39853): tablet 0d6348ffe8e74d03a85353dd311c20c2 (table test-table [id=c3bdfe2932154dffba642f9a4dd04447]) successfully deleted
W20260619 02:25:15.609689 7891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41619 (error 108)
I20260619 02:25:15.610579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 8118
I20260619 02:25:15.610817 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:15.610388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41619 (local address 127.0.0.1:60164)
0619 02:25:15.610600 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:15.610693 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
W20260619 02:25:15.612155 7891 catalog_manager.cc:4584] TS c95ae6aba17f40efab8223fe639054d2 (127.0.98.1:41619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d6348ffe8e74d03a85353dd311c20c2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41619: connect: Connection refused (error 111)
I20260619 02:25:15.617614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 8258
I20260619 02:25:15.623795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 7874
2026-06-19T02:25:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3444 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:25:15 dist-test-slave-xkrt krb5kdc[8511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:25:15 dist-test-slave-xkrt krb5kdc[8511](info): set up 2 sockets
Jun 19 02:25:15 dist-test-slave-xkrt krb5kdc[8511](info): commencing operation
krb5kdc: starting...
W20260619 02:25:16.738457 392 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:25:16 dist-test-slave-xkrt krb5kdc[8511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:25:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:25:16.857285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43083
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:16.964839 8524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:16.965121 8524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:16.965198 8524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:16.969161 8524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:16.969236 8524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:16.969273 8524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:16.969331 8524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:16.969375 8524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:25:16.974426 8524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:16.975401 8524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:16.976356 8524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:16.984619 8529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:16.984699 8524 server_base.cc:1034] running on GCE node
W20260619 02:25:16.984741 8530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:16.984627 8532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:16.985473 8524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:16.986575 8524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:16.987788 8524 hybrid_clock.cc:648] HybridClock initialized: now 1781835916987753 us; error 48 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T02:25:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1369 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:25:17 dist-test-slave-xkrt krb5kdc[8537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:25:17 dist-test-slave-xkrt krb5kdc[8537](info): set up 2 sockets
Jun 19 02:25:17 dist-test-slave-xkrt krb5kdc[8537](info): commencing operation
krb5kdc: starting...
W20260619 02:25:18.038303 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:25:18 dist-test-slave-xkrt krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:25:18.157059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35451
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:18.266310 8550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:18.266549 8550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:18.266587 8550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:18.270498 8550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:18.270550 8550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:18.270568 8550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:18.270586 8550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:18.270601 8550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:25:18.275605 8550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:18.276499 8550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:18.277415 8550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:18.285768 8558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:18.285791 8556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:18.285784 8555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:18.285990 8550 server_base.cc:1034] running on GCE node
I20260619 02:25:18.286571 8550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:18.287668 8550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:18.288838 8550 hybrid_clock.cc:648] HybridClock initialized: now 1781835918288820 us; error 32 us; skew 500 ppm
Jun 19 02:25:18 dist-test-slave-xkrt krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:18.291514 8550 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:25:18.292337 8550 webserver.cc:460] Webserver started at http://127.0.98.62:46129/ using document root <none> and password file <none>
I20260619 02:25:18.292882 8550 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:18.292972 8550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:18.293195 8550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:18.295213 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "2a7f9e0e4abe497283e113bcd28de81e"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "d2ede5847114a8d35878b419e6e32057"
server_key_iv: "3194188e5953b6de9cb0b1ea60a45663"
server_key_version: "encryptionkey@0"
I20260619 02:25:18.295789 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a7f9e0e4abe497283e113bcd28de81e"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "d2ede5847114a8d35878b419e6e32057"
server_key_iv: "3194188e5953b6de9cb0b1ea60a45663"
server_key_version: "encryptionkey@0"
I20260619 02:25:18.299310 8550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 02:25:18.301714 8565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:18.302953 8550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:18.303112 8550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "2a7f9e0e4abe497283e113bcd28de81e"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "d2ede5847114a8d35878b419e6e32057"
server_key_iv: "3194188e5953b6de9cb0b1ea60a45663"
server_key_version: "encryptionkey@0"
I20260619 02:25:18.303220 8550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:18.329896 8550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:18.332671 8550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:18.332871 8550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:18.339849 8550 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35451
I20260619 02:25:18.339876 8617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35451 every 8 connection(s)
I20260619 02:25:18.340889 8550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:25:18.342760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 8550
I20260619 02:25:18.342890 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:25:18.343164 392 external_mini_cluster.cc:1428] Setting key f8c7cfae5b3e82f972529e33ccc90a7d
I20260619 02:25:18.343722 8618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:18.349288 8618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e: Bootstrap starting.
Jun 19 02:25:18 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:25:18.352015 8618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:18.352923 8618 log.cc:824] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:18.355190 8618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e: No bootstrap required, opened a new log
I20260619 02:25:18.357178 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:18.344504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.0.98.62:35451)
0619 02:25:18.344925 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:18.344942 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:18.344975 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:25:18.345679 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:18.345696 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:18.346009 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:18.346326 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:18.347091 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:18.348066 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:18.348922 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:18.349389 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:18.349601 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:18.352264 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:18.352273 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:18.352293 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:18.352335 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:18.354214 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:18.354643 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:18.354649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:18.354655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:18.354783 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:18.355042 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:18.355045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:18.355047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:18.355536 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:18.355724 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:18.355989 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 02:25:18.356267 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":132,"threads_started":1}
I20260619 02:25:18.359449 8618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7f9e0e4abe497283e113bcd28de81e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35451 } }
I20260619 02:25:18.359663 8618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:18.359696 8618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7f9e0e4abe497283e113bcd28de81e, State: Initialized, Role: FOLLOWER
I20260619 02:25:18.360083 8618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7f9e0e4abe497283e113bcd28de81e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35451 } }
I20260619 02:25:18.360198 8618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:25:18.360235 8618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:25:18.360302 8618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:18.361096 8618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7f9e0e4abe497283e113bcd28de81e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35451 } }
I20260619 02:25:18.361375 8618 leader_election.cc:302] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a7f9e0e4abe497283e113bcd28de81e; no voters:
I20260619 02:25:18.361696 8618 leader_election.cc:288] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:25:18.361853 8623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:18.362111 8623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 1 LEADER]: Becoming Leader. State: Replica: 2a7f9e0e4abe497283e113bcd28de81e, State: Running, Role: LEADER
I20260619 02:25:18.362701 8623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7f9e0e4abe497283e113bcd28de81e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35451 } }
I20260619 02:25:18.363215 8618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:18.364501 8624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a7f9e0e4abe497283e113bcd28de81e. Latest consensus state: current_term: 1 leader_uuid: "2a7f9e0e4abe497283e113bcd28de81e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7f9e0e4abe497283e113bcd28de81e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35451 } } }
I20260619 02:25:18.364646 8624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:18.364486 8625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a7f9e0e4abe497283e113bcd28de81e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7f9e0e4abe497283e113bcd28de81e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35451 } } }
I20260619 02:25:18.364886 8625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:18.365154 8632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:18.368283 8632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:18.373865 8632 catalog_manager.cc:1347] Generated new cluster ID: 6f506f830e44426ba80b6885c15d31e7
I20260619 02:25:18.373925 8632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:18.379854 8632 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:25:18.380988 8632 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:18.388870 8632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e: Generated new TSK 0
I20260619 02:25:18.389504 8632 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:25:18.428471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35451
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:18.535815 8644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:18.536093 8644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:18.536168 8644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:18.540156 8644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:18.540239 8644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:18.540331 8644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:25:18.547996 8644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:18.549127 8644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:18.550236 8644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:18.559636 8650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:18.559670 8652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:18.559702 8649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:18.559927 8644 server_base.cc:1034] running on GCE node
I20260619 02:25:18.560349 8644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:18.560976 8644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:18.562177 8644 hybrid_clock.cc:648] HybridClock initialized: now 1781835918562159 us; error 30 us; skew 500 ppm
Jun 19 02:25:18 dist-test-slave-xkrt krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:18.565241 8644 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:25:18.566206 8644 webserver.cc:460] Webserver started at http://127.0.98.1:41853/ using document root <none> and password file <none>
I20260619 02:25:18.566754 8644 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:18.566840 8644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:18.567052 8644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:18.568996 8644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "bbf23ce30deb4e96b3a2f94c008c0fce"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "d15c7ba6e13e0a7a64db9c80a586b5b9"
server_key_iv: "8a78887b42509e728068e75d812a578d"
server_key_version: "encryptionkey@0"
I20260619 02:25:18.569532 8644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bbf23ce30deb4e96b3a2f94c008c0fce"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "d15c7ba6e13e0a7a64db9c80a586b5b9"
server_key_iv: "8a78887b42509e728068e75d812a578d"
server_key_version: "encryptionkey@0"
I20260619 02:25:18.573089 8644 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:25:18.575647 8659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:18.576895 8644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:25:18.577039 8644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "bbf23ce30deb4e96b3a2f94c008c0fce"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "d15c7ba6e13e0a7a64db9c80a586b5b9"
server_key_iv: "8a78887b42509e728068e75d812a578d"
server_key_version: "encryptionkey@0"
I20260619 02:25:18.577150 8644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:18.588929 8644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:18.591728 8644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:18.591924 8644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:18.592485 8644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:18.593257 8644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:18.593328 8644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:18.593395 8644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:18.593452 8644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:18.602384 8644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40495
I20260619 02:25:18.602416 8772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40495 every 8 connection(s)
I20260619 02:25:18.603225 8644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:25:18.604270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 8644
I20260619 02:25:18.604411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:25:18.604713 392 external_mini_cluster.cc:1428] Setting key fb76518ccb1420504ef1b6aa8fac9f93
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 19 02:25:18 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 02:25:18.618258 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:18.605316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34051 (local address 127.0.98.62:35451)
0619 02:25:18.605482 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:18.605486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:18.606516 (+ 1030us) server_negotiation.cc:407] Connection header received
0619 02:25:18.608048 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:18.608053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:18.608112 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:18.608203 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:18.609459 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:18.610074 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:18.611292 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:18.611546 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:18.611657 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:18.614051 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:18.614055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:18.614057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:18.614082 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:18.615687 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:18.616301 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:18.616307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:18.616356 (+ 49us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:18.616448 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:18.616898 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:18.616901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:18.616902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:18.617072 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:18.617148 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:18.617905 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 02:25:18.618043 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:25:18.618945 8775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:18.605841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:34051)
0619 02:25:18.606354 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:18.606406 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:18.607785 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:18.608385 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:18.608394 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:18.608837 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:18.609318 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:18.609331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:18.610199 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:18.610205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:18.611176 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:18.611185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:18.611693 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:18.611696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:18.611867 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:18.612465 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:18.612482 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:18.613892 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:18.615819 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:18.615824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:18.615835 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:18.616190 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:18.616577 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:18.616581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:18.616583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:18.616788 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:18.617177 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:18.617186 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:18.617581 (+ 395us) client_negotiation.cc:769] Sending connection context
0619 02:25:18.617830 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 02:25:18.618093 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":162,"threads_started":1}
I20260619 02:25:18.620193 8773 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35451
I20260619 02:25:18.620465 8773 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:18.621055 8773 heartbeater.cc:511] Master 127.0.98.62:35451 requested a full tablet report, sending...
I20260619 02:25:18.622797 8582 ts_manager.cc:194] Registered new tserver with Master: bbf23ce30deb4e96b3a2f94c008c0fce (127.0.98.1:40495)
I20260619 02:25:18.624431 8582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:25:18.632863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35451
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:18.744961 8778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:18.745204 8778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:18.745245 8778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:18.749229 8778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:18.749282 8778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:18.749370 8778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:25:18.754432 8778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:18.755390 8778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:18.756459 8778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:18.765812 8786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:18.765831 8783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:18.765831 8784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:18.765899 8778 server_base.cc:1034] running on GCE node
I20260619 02:25:18.766587 8778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:18.767251 8778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:18.768461 8778 hybrid_clock.cc:648] HybridClock initialized: now 1781835918768446 us; error 32 us; skew 500 ppm
Jun 19 02:25:18 dist-test-slave-xkrt krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:18.771162 8778 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:25:18.772035 8778 webserver.cc:460] Webserver started at http://127.0.98.2:37175/ using document root <none> and password file <none>
I20260619 02:25:18.772585 8778 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:18.772686 8778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:18.772908 8778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:18.774966 8778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "14f5ccfcf3dd4ba1a722265bd8097c19"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "ee1423a5e1abb418b8e90c3c1dd16dfb"
server_key_iv: "e54766624155f7cc6a55ab837ca03852"
server_key_version: "encryptionkey@0"
I20260619 02:25:18.775506 8778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "14f5ccfcf3dd4ba1a722265bd8097c19"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "ee1423a5e1abb418b8e90c3c1dd16dfb"
server_key_iv: "e54766624155f7cc6a55ab837ca03852"
server_key_version: "encryptionkey@0"
I20260619 02:25:18.779156 8778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:25:18.781697 8793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:18.782895 8778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:25:18.783026 8778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "14f5ccfcf3dd4ba1a722265bd8097c19"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "ee1423a5e1abb418b8e90c3c1dd16dfb"
server_key_iv: "e54766624155f7cc6a55ab837ca03852"
server_key_version: "encryptionkey@0"
I20260619 02:25:18.783135 8778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:18.792943 8778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:18.795852 8778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:18.796046 8778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:18.796583 8778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:18.797366 8778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:18.797435 8778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:18.797495 8778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:18.797537 8778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:18.806362 8778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44471
I20260619 02:25:18.806392 8906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44471 every 8 connection(s)
I20260619 02:25:18.807260 8778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:25:18.808653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 8778
I20260619 02:25:18.808800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:25:18.809144 392 external_mini_cluster.cc:1428] Setting key c43e098fcb819e3292c3261637fb47d1
Jun 19 02:25:18 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 02:25:18.824088 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:18.808998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59333 (local address 127.0.98.62:35451)
0619 02:25:18.809148 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:18.809154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:18.810223 (+ 1069us) server_negotiation.cc:407] Connection header received
0619 02:25:18.812401 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:18.812404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:18.812444 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:18.812515 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:18.813922 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:18.814442 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:18.816447 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:18.816663 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:18.816747 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:18.819948 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:18.819951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:18.819954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:18.819977 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:18.821738 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:18.822253 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:18.822256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:18.822258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:18.822412 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:18.822875 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:18.822878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:18.822879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:18.823069 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:18.823188 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:18.823749 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 02:25:18.823899 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:25:18.825169 8909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:18.809458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:59333)
0619 02:25:18.810076 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:18.810120 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:18.812159 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:18.812710 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:18.812722 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:18.813144 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:18.813735 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:18.813776 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:18.814575 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:18.814582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:18.816315 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:18.816331 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:18.816846 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:18.816854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:18.817106 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:18.817832 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:18.817857 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:18.819738 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:18.821894 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:18.821898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:18.821909 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:18.822150 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:18.822546 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:18.822549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:18.822552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:18.822757 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:18.823189 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:18.823197 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:18.823486 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 02:25:18.823688 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 02:25:18.824036 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":142,"threads_started":1}
I20260619 02:25:18.826480 8907 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35451
I20260619 02:25:18.826732 8907 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:18.827210 8907 heartbeater.cc:511] Master 127.0.98.62:35451 requested a full tablet report, sending...
I20260619 02:25:18.828260 8582 ts_manager.cc:194] Registered new tserver with Master: 14f5ccfcf3dd4ba1a722265bd8097c19 (127.0.98.2:44471)
I20260619 02:25:18.828823 8582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:25:18.840946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35451
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:18.955015 8912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:18.955248 8912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:18.955288 8912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:18.960124 8912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:18.960181 8912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:18.960237 8912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:25:18.965744 8912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:18.966809 8912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:18.967721 8912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:18.976843 8918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:18.976858 8920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:18.977123 8912 server_base.cc:1034] running on GCE node
W20260619 02:25:18.976850 8917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:18.977511 8912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:18.978147 8912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:18.979305 8912 hybrid_clock.cc:648] HybridClock initialized: now 1781835918979273 us; error 43 us; skew 500 ppm
Jun 19 02:25:18 dist-test-slave-xkrt krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:18.982105 8912 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:25:18.982844 8912 webserver.cc:460] Webserver started at http://127.0.98.3:36225/ using document root <none> and password file <none>
I20260619 02:25:18.983337 8912 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:18.983392 8912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:18.983541 8912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:18.985357 8912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "1c33cbdc50b144aebb2a6b867f5fd616"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "1fdfa98d6456a12f671dd9b6361068c0"
server_key_iv: "533648414a789cde52fef44e71953a12"
server_key_version: "encryptionkey@0"
I20260619 02:25:18.986017 8912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1c33cbdc50b144aebb2a6b867f5fd616"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "1fdfa98d6456a12f671dd9b6361068c0"
server_key_iv: "533648414a789cde52fef44e71953a12"
server_key_version: "encryptionkey@0"
I20260619 02:25:18.989594 8912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:25:18.991773 8927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:18.992946 8912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:18.993063 8912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "1c33cbdc50b144aebb2a6b867f5fd616"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "1fdfa98d6456a12f671dd9b6361068c0"
server_key_iv: "533648414a789cde52fef44e71953a12"
server_key_version: "encryptionkey@0"
I20260619 02:25:18.993160 8912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:19.006186 8912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:19.009322 8912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:19.009510 8912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:19.010118 8912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:19.010881 8912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:19.010927 8912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:19.010964 8912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:19.011005 8912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:19.019874 8912 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36983
I20260619 02:25:19.019899 9040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36983 every 8 connection(s)
I20260619 02:25:19.020677 8912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:25:19.026877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 8912
I20260619 02:25:19.026973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:25:19.027251 392 external_mini_cluster.cc:1428] Setting key 35f583a74e7c8b054d37f39c1c3a42ea
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:25:19.034087 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:19.022471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54553 (local address 127.0.98.62:35451)
0619 02:25:19.022606 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:19.022610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:19.023425 (+ 815us) server_negotiation.cc:407] Connection header received
0619 02:25:19.024813 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:19.024817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:19.024872 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:19.024960 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:19.026146 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.026676 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.028074 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.028416 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.028512 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.030951 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:19.030954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:19.030957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:19.030980 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:19.032331 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:19.032798 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:19.032801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:19.032802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:19.032875 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:19.033160 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:19.033162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:19.033163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:19.033326 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:19.033395 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:19.033885 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 02:25:19.033977 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:25:19.034876 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:19.022800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:54553)
0619 02:25:19.023241 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:19.023293 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:19.024609 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:19.025134 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:19.025140 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:19.025469 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:19.025993 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.026006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.026807 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.026812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.027958 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.027970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.028558 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.028566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.028723 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.029349 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:19.029379 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:19.030806 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:19.032449 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:19.032453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:19.032463 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:19.032705 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:19.032958 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:19.032961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:19.032962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:19.033082 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:19.033409 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:19.033413 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:19.033691 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 02:25:19.033843 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 02:25:19.034073 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":120,"threads_started":1}
I20260619 02:25:19.035938 9041 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35451
I20260619 02:25:19.036118 9041 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:19.036553 9041 heartbeater.cc:511] Master 127.0.98.62:35451 requested a full tablet report, sending...
I20260619 02:25:19.037531 8582 ts_manager.cc:194] Registered new tserver with Master: 1c33cbdc50b144aebb2a6b867f5fd616 (127.0.98.3:36983)
I20260619 02:25:19.038113 8582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54553
I20260619 02:25:19.040892 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 02:25:19.111410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 8644
I20260619 02:25:19.116845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 8778
I20260619 02:25:19.121680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 8912
I20260619 02:25:19.126678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 8550
I20260619 02:25:19.132504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35451
--webserver_interface=127.0.98.62
--webserver_port=46129
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:19.239612 9050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:19.239899 9050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:19.239984 9050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:19.243959 9050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:19.244051 9050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:19.244091 9050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:19.244151 9050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:19.244190 9050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:25:19.249348 9050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:19.250429 9050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:19.251425 9050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:19.259819 9056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:19.259878 9055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:19.259858 9050 server_base.cc:1034] running on GCE node
W20260619 02:25:19.260037 9058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:19.260520 9050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:19.261613 9050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:19.262809 9050 hybrid_clock.cc:648] HybridClock initialized: now 1781835919262776 us; error 52 us; skew 500 ppm
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:19.265518 9050 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:25:19.266405 9050 webserver.cc:460] Webserver started at http://127.0.98.62:46129/ using document root <none> and password file <none>
I20260619 02:25:19.266952 9050 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:19.267053 9050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:19.270735 9050 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 02:25:19.272701 9065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:19.273870 9050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:25:19.274006 9050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "2a7f9e0e4abe497283e113bcd28de81e"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "d2ede5847114a8d35878b419e6e32057"
server_key_iv: "3194188e5953b6de9cb0b1ea60a45663"
server_key_version: "encryptionkey@0"
I20260619 02:25:19.274505 9050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:19.288066 9050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:19.290697 9050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:19.290894 9050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:19.297873 9050 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35451
I20260619 02:25:19.297947 9117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35451 every 8 connection(s)
I20260619 02:25:19.298915 9050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:25:19.302989 9118 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:25:19.304522 9118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e: Bootstrap starting.
I20260619 02:25:19.307977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 9050
I20260619 02:25:19.308471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40495
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35451
--webserver_port=41853
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:25:19.321223 9118 log.cc:824] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:19.327961 9118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:25:19.328329 9118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e: Bootstrap complete.
I20260619 02:25:19.332443 9118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7f9e0e4abe497283e113bcd28de81e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35451 } }
I20260619 02:25:19.332814 9118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7f9e0e4abe497283e113bcd28de81e, State: Initialized, Role: FOLLOWER
I20260619 02:25:19.333292 9118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7f9e0e4abe497283e113bcd28de81e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35451 } }
I20260619 02:25:19.333441 9118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:25:19.333505 9118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:25:19.333644 9118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:25:19.335492 9118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7f9e0e4abe497283e113bcd28de81e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35451 } }
I20260619 02:25:19.335816 9118 leader_election.cc:302] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a7f9e0e4abe497283e113bcd28de81e; no voters:
I20260619 02:25:19.336136 9118 leader_election.cc:288] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:25:19.336221 9122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:25:19.336400 9122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [term 2 LEADER]: Becoming Leader. State: Replica: 2a7f9e0e4abe497283e113bcd28de81e, State: Running, Role: LEADER
I20260619 02:25:19.336678 9122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7f9e0e4abe497283e113bcd28de81e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35451 } }
I20260619 02:25:19.337325 9118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:19.337949 9123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2a7f9e0e4abe497283e113bcd28de81e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7f9e0e4abe497283e113bcd28de81e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35451 } } }
I20260619 02:25:19.338066 9123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:19.338281 9123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a7f9e0e4abe497283e113bcd28de81e. Latest consensus state: current_term: 2 leader_uuid: "2a7f9e0e4abe497283e113bcd28de81e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7f9e0e4abe497283e113bcd28de81e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35451 } } }
I20260619 02:25:19.338378 9123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:19.339318 9127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:19.342469 9127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:19.343076 9127 catalog_manager.cc:1259] Loaded cluster ID: 6f506f830e44426ba80b6885c15d31e7
I20260619 02:25:19.343135 9127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:19.345407 9127 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:19.346064 9127 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2a7f9e0e4abe497283e113bcd28de81e: Loaded TSK: 0
I20260619 02:25:19.347227 9127 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:25:19.418388 9120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:19.418648 9120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:19.418722 9120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:19.422634 9120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:19.422715 9120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:19.422796 9120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:25:19.427896 9120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41853
--tserver_master_addrs=127.0.98.62:35451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:19.428812 9120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:19.429818 9120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:19.439951 9145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:19.439939 9147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:19.439939 9144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:19.440506 9120 server_base.cc:1034] running on GCE node
I20260619 02:25:19.440887 9120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:19.441488 9120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:19.442698 9120 hybrid_clock.cc:648] HybridClock initialized: now 1781835919442670 us; error 48 us; skew 500 ppm
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835919, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:19.445513 9120 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 02:25:19.446412 9120 webserver.cc:460] Webserver started at http://127.0.98.1:41853/ using document root <none> and password file <none>
I20260619 02:25:19.446949 9120 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:19.447049 9120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:19.450816 9120 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260619 02:25:19.452963 9154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:19.454263 9120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:25:19.454399 9120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "bbf23ce30deb4e96b3a2f94c008c0fce"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "d15c7ba6e13e0a7a64db9c80a586b5b9"
server_key_iv: "8a78887b42509e728068e75d812a578d"
server_key_version: "encryptionkey@0"
I20260619 02:25:19.454923 9120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:19.475039 9120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:19.478206 9120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:19.478396 9120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:19.479001 9120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:19.479843 9120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:19.479910 9120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:19.479993 9120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:19.480034 9120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:19.488761 9120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40495
I20260619 02:25:19.488790 9267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40495 every 8 connection(s)
I20260619 02:25:19.489749 9120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:25:19.494663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 9120
I20260619 02:25:19.495162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44471
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35451
--webserver_port=37175
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:19.502949 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:19.491822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:33103)
0619 02:25:19.492266 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:19.492304 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:19.493495 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:19.494633 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:19.494639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:19.495030 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:19.495556 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.495569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.497131 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.497138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.498230 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.498239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.499356 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.499360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.499491 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.500077 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:19.500092 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:19.502267 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":169,"threads_started":1}
W20260619 02:25:19.503418 9268 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35451 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35451: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260619 02:25:19.503908 9270 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:19.491711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33103 (local address 127.0.98.62:35451)
0619 02:25:19.492138 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:19.492157 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:19.492459 (+ 302us) server_negotiation.cc:407] Connection header received
0619 02:25:19.493907 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:19.493918 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:19.494191 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:19.494595 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:19.495721 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.497003 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.498364 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.499178 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.499384 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.502565 (+ 3181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33103: BlockingRecv error: recv got EOF from 127.0.98.1:33103 (error 108)
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":90,"threads_started":1}
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:19.508884 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:19.503847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:51335)
0619 02:25:19.504001 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:19.504018 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:19.504091 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:19.504554 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:19.504557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:19.504702 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:19.504778 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.504782 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.505482 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.505488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.506062 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.506070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.506519 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.506522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.506613 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.507073 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:19.507084 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:19.508770 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260619 02:25:19.509066 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:19.503918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51335 (local address 127.0.98.62:35451)
0619 02:25:19.504362 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:19.504369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:19.504379 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:25:19.504408 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:19.504410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:19.504443 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:19.504541 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:19.504886 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.505382 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.506172 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.506406 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.506509 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.508891 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51335: BlockingRecv error: recv got EOF from 127.0.98.1:51335 (error 108)
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":123,"threads_started":1}
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:19.513965 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:19.509403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:52785)
0619 02:25:19.509506 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:19.509521 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:19.509607 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:19.509817 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:19.509820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:19.509960 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:19.510019 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.510023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.510689 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.510694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.511192 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.511199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.511620 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.511623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.511697 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.512144 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:19.512156 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:19.513837 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 02:25:19.514155 9268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:25:19.514281 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:19.509497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52785 (local address 127.0.98.62:35451)
0619 02:25:19.509617 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:19.509621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:19.509636 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:25:19.509674 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:19.509677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:19.509718 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:19.509811 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:19.510122 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.510591 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.511301 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.511516 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.511620 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.513946 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52785: BlockingRecv error: recv got EOF from 127.0.98.1:52785 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 02:25:19.606014 9272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:19.606266 9272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:19.606354 9272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:19.610436 9272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:19.610488 9272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:19.610580 9272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:25:19.615795 9272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37175
--tserver_master_addrs=127.0.98.62:35451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:19.616859 9272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:19.617940 9272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:19.627398 9278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:19.627480 9281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:19.627449 9279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:19.627628 9272 server_base.cc:1034] running on GCE node
I20260619 02:25:19.628006 9272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:19.628613 9272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:19.629794 9272 hybrid_clock.cc:648] HybridClock initialized: now 1781835919629759 us; error 43 us; skew 500 ppm
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835919, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:19.632444 9272 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 02:25:19.633303 9272 webserver.cc:460] Webserver started at http://127.0.98.2:37175/ using document root <none> and password file <none>
I20260619 02:25:19.633903 9272 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:19.634006 9272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:19.637708 9272 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:25:19.639714 9288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:19.640885 9272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:19.641017 9272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "14f5ccfcf3dd4ba1a722265bd8097c19"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "ee1423a5e1abb418b8e90c3c1dd16dfb"
server_key_iv: "e54766624155f7cc6a55ab837ca03852"
server_key_version: "encryptionkey@0"
I20260619 02:25:19.641546 9272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:19.657035 9272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:19.660113 9272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:19.660327 9272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:19.660905 9272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:19.661795 9272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:19.661871 9272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:19.661969 9272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:19.662027 9272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:19.671614 9272 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44471
I20260619 02:25:19.671650 9401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44471 every 8 connection(s)
I20260619 02:25:19.672695 9272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:25:19.681449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 9272
I20260619 02:25:19.682029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36983
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35451
--webserver_port=36225
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:25:19.684396 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:19.674437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40815 (local address 127.0.98.62:35451)
0619 02:25:19.674588 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:19.674593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:19.675461 (+ 868us) server_negotiation.cc:407] Connection header received
0619 02:25:19.676897 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:19.676901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:19.676964 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:19.677050 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:19.678330 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.678918 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.680145 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.680398 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.680521 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.684242 (+ 3721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40815: BlockingRecv error: recv got EOF from 127.0.98.2:40815 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:25:19.684829 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:19.674840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:40815)
0619 02:25:19.675259 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:19.675301 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:19.676676 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:19.677218 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:19.677225 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:19.677652 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:19.678187 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.678200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.679065 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.679070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.680010 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.680019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.680565 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.680570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.680741 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.681374 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:19.681400 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:19.684062 (+ 2662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":112,"threads_started":1}
W20260619 02:25:19.685322 9402 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35451 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35451: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:19.690529 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:19.685797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:53143)
0619 02:25:19.685882 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:19.685897 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:19.685975 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:19.686197 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:19.686200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:19.686344 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:19.686421 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.686426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.687130 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.687136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.687676 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.687683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.688124 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.688127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.688218 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.688679 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:19.688691 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:19.690394 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 02:25:19.690620 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:19.685878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53143 (local address 127.0.98.62:35451)
0619 02:25:19.685981 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:19.685984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:19.685999 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:25:19.686044 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:19.686048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:19.686095 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:19.686173 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:19.686538 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.687022 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.687789 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.688004 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.688121 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.690510 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53143: BlockingRecv error: recv got EOF from 127.0.98.2:53143 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:19.695771 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:19.691254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:44781)
0619 02:25:19.691358 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:19.691373 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:19.691443 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:19.691684 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:19.691687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:19.691831 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:19.691890 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.691895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.692557 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.692563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.693068 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.693075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.693493 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.693497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.693599 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.694046 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:19.694056 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:19.695659 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 02:25:19.695875 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:19.691363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44781 (local address 127.0.98.62:35451)
0619 02:25:19.691499 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:19.691503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:19.691516 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:25:19.691555 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:19.691557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:19.691591 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:19.691673 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:19.691999 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.692458 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.693179 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.693380 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.693485 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.695766 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44781: BlockingRecv error: recv got EOF from 127.0.98.2:44781 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 02:25:19.695993 9402 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:25:19.791437 9405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:19.791672 9405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:19.791712 9405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:19.795617 9405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:19.795672 9405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:19.795727 9405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:25:19.800773 9405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36225
--tserver_master_addrs=127.0.98.62:35451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:19.801712 9405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:19.802693 9405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:19.811928 9411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:19.812012 9410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:19.812135 9413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:19.811990 9405 server_base.cc:1034] running on GCE node
I20260619 02:25:19.812512 9405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:19.813143 9405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:19.814320 9405 hybrid_clock.cc:648] HybridClock initialized: now 1781835919814307 us; error 26 us; skew 500 ppm
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835919, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:19.817009 9405 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 02:25:19.817911 9405 webserver.cc:460] Webserver started at http://127.0.98.3:36225/ using document root <none> and password file <none>
I20260619 02:25:19.818460 9405 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:19.818565 9405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:19.822341 9405 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:19.824373 9420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:19.825635 9405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:19.825757 9405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "1c33cbdc50b144aebb2a6b867f5fd616"
format_stamp: "Formatted at 2026-06-19 02:25:18 on dist-test-slave-xkrt"
server_key: "1fdfa98d6456a12f671dd9b6361068c0"
server_key_iv: "533648414a789cde52fef44e71953a12"
server_key_version: "encryptionkey@0"
I20260619 02:25:19.826288 9405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:19.856246 9405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:19.859714 9405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:19.859910 9405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:19.860503 9405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:19.861325 9405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:19.861394 9405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:19.861466 9405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:19.861521 9405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:19.870759 9405 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36983
I20260619 02:25:19.870790 9533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36983 every 8 connection(s)
I20260619 02:25:19.871721 9405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:25:19.879112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 9405
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:25:19.882970 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:19.873429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48683 (local address 127.0.98.62:35451)
0619 02:25:19.873581 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:19.873586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:19.874401 (+ 815us) server_negotiation.cc:407] Connection header received
0619 02:25:19.875692 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:19.875695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:19.875744 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:19.875818 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:19.876917 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.877488 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.878691 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.878924 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.879009 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.882837 (+ 3828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48683: BlockingRecv error: recv got EOF from 127.0.98.3:48683 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:25:19.883502 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:19.873822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:48683)
0619 02:25:19.874247 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:19.874287 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:19.875499 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:19.875981 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:19.875988 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:19.876316 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:19.876792 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.876804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.877629 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.877632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.878561 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.878569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.879097 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.879102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.879331 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.879979 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:19.880000 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:19.882696 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
W20260619 02:25:19.884022 9534 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35451 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35451: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:19.889623 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:19.880291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.0.98.62:35451)
0619 02:25:19.880446 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:19.880450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:19.880464 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:25:19.880500 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:19.880506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:19.880557 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:19.880659 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:19.881508 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.882264 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.883356 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.883571 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.883675 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.885393 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:19.885399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:19.885412 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:19.885437 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:19.887291 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:19.887880 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:19.887885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:19.887889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:19.888007 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:19.888345 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:19.888347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:19.888348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:19.888728 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:19.888942 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:19.889222 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 02:25:19.889466 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 02:25:19.889685 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:19.884433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:49779)
0619 02:25:19.884540 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:19.884554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:19.884622 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:19.884897 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:19.884900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:19.885037 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:19.885124 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.885129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.885929 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.885934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.886467 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.886474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.886905 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.886908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.887007 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.887458 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:19.887474 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:19.889523 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 02:25:19.889776 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:19.884538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49779 (local address 127.0.98.62:35451)
0619 02:25:19.884642 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:19.884647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:19.884680 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:25:19.884717 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:19.884721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:19.884763 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:19.884853 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:19.885236 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.885827 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.886590 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.886796 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.886901 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.889663 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49779: BlockingRecv error: recv got EOF from 127.0.98.3:49779 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:19 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:19.895516 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:19.890909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:45289)
0619 02:25:19.891048 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:19.891061 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:19.891125 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:19.891369 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:19.891372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:19.891528 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:19.891618 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.891625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.892514 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.892517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.893010 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:19.893017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.893387 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.893389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:19.893448 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.893906 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:19.893925 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:19.895435 (+ 1510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83}
W20260619 02:25:19.895584 9270 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:19.890998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45289 (local address 127.0.98.62:35451)
0619 02:25:19.891120 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:19.891123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:19.891134 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:25:19.891201 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:19.891203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:19.891255 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:19.891337 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:19.891735 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.892405 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.893102 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:19.893299 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:19.893368 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:19.895515 (+ 2147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45289: BlockingRecv error: recv got EOF from 127.0.98.3:45289 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 02:25:19.895727 9534 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 02:25:20 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:20 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:20.520042 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:20.514712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:42433)
0619 02:25:20.514980 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:20.514994 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:20.515080 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:20.515355 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:20.515357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:20.515568 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:20.515680 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:20.515686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:20.516487 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:20.516489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:20.517053 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:20.517061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:20.517460 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:20.517462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:20.517531 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:20.518164 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:20.518176 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:20.519914 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
W20260619 02:25:20.520228 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:20.514790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42433 (local address 127.0.98.62:35451)
0619 02:25:20.515031 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:20.515034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:20.515046 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:25:20.515184 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:20.515187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:20.515246 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:20.515316 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:20.515781 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:20.516365 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:20.517151 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:20.517367 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:20.517442 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:20.520078 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42433: BlockingRecv error: recv got EOF from 127.0.98.1:42433 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":93,"threads_started":1}
Jun 19 02:25:20 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:20 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:20.702596 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:20.696710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:41967)
0619 02:25:20.697002 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:20.697017 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:20.697119 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:20.697362 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:20.697365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:20.697578 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:20.697694 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:20.697700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:20.698649 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:20.698652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:20.699432 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:20.699442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:20.699979 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:20.699983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:20.700060 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:20.700660 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:20.700675 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:20.702479 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":65,"threads_started":1}
W20260619 02:25:20.702732 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:20.696777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41967 (local address 127.0.98.62:35451)
0619 02:25:20.696975 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:20.696979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:20.697090 (+ 111us) server_negotiation.cc:407] Connection header received
0619 02:25:20.697199 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:20.697202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:20.697243 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:20.697309 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:20.697815 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:20.698514 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:20.699632 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:20.699846 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:20.699932 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:20.702619 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41967: BlockingRecv error: recv got EOF from 127.0.98.2:41967 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Jun 19 02:25:20 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:20 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:20.901531 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:20.896313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:35461)
0619 02:25:20.896604 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:20.896618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:20.896715 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:20.896950 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:20.896952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:20.897125 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:20.897232 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:20.897237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:20.897951 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:20.897954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:20.898553 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:20.898562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:20.898964 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:20.898967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:20.899040 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:20.899602 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:20.899614 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:20.901409 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":92,"threads_started":1}
W20260619 02:25:20.901661 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:20.896412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35461 (local address 127.0.98.62:35451)
0619 02:25:20.896591 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:20.896595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:20.896686 (+ 91us) server_negotiation.cc:407] Connection header received
0619 02:25:20.896796 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:20.896799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:20.896843 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:20.896908 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:20.897327 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:20.897860 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:20.898655 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:20.898864 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:20.898937 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:20.901571 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35461: BlockingRecv error: recv got EOF from 127.0.98.3:35461 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 19 02:25:21 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:21 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:21.526559 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:21.520875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:54911)
0619 02:25:21.521150 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:21.521168 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:21.521270 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:21.521527 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:21.521529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:21.521720 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:21.521810 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:21.521815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:21.522625 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:21.522627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:21.523182 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:21.523189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:21.523657 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:21.523659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:21.523723 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:21.524259 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:21.524273 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:21.526408 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
W20260619 02:25:21.526638 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:21.521021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54911 (local address 127.0.98.62:35451)
0619 02:25:21.521264 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:21.521268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:21.521281 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:25:21.521350 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:21.521353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:21.521398 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:21.521466 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:21.521940 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:21.522522 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:21.523302 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:21.523556 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:21.523647 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:21.526536 (+ 2889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54911: BlockingRecv error: recv got EOF from 127.0.98.1:54911 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
Jun 19 02:25:21 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:21 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:21.708727 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:21.703367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:43839)
0619 02:25:21.703644 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:21.703657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:21.703752 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:21.703955 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:21.703957 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:21.704134 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:21.704225 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:21.704230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:21.704984 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:21.704987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:21.705737 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:21.705765 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:21.706308 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:21.706311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:21.706377 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:21.706881 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:21.706891 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:21.708602 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":112,"threads_started":1}
W20260619 02:25:21.708844 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:21.703422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43839 (local address 127.0.98.62:35451)
0619 02:25:21.703563 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:21.703567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:21.703725 (+ 158us) server_negotiation.cc:407] Connection header received
0619 02:25:21.703817 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:21.703819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:21.703861 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:21.703932 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:21.704316 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:21.704855 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:21.705875 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:21.706204 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:21.706291 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:21.708731 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43839: BlockingRecv error: recv got EOF from 127.0.98.2:43839 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 02:25:21 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:21 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:21.907500 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:21.902367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:36025)
0619 02:25:21.902639 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:21.902652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:21.902754 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:21.902977 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:21.902980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:21.903153 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:21.903248 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:21.903253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:21.903993 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:21.903996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:21.904568 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:21.904578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:21.905025 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:21.905028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:21.905093 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:21.905593 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:21.905605 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:21.907364 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":120,"threads_started":1}
W20260619 02:25:21.907612 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:21.902495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36025 (local address 127.0.98.62:35451)
0619 02:25:21.902621 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:21.902625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:21.902715 (+ 90us) server_negotiation.cc:407] Connection header received
0619 02:25:21.902830 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:21.902832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:21.902875 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:21.902940 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:21.903350 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:21.903870 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:21.904675 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:21.904921 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:21.905001 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:21.907501 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36025: BlockingRecv error: recv got EOF from 127.0.98.3:36025 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 02:25:22 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:22 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:22.536322 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:22.527329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:32923)
0619 02:25:22.527684 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:22.527698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:22.527793 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:22.528052 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:22.528054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:22.528229 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:22.528326 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:22.528331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:22.529115 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:22.529118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:22.529735 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:22.529744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:22.530218 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:22.530222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:22.530303 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:22.534413 (+ 4110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:22.534426 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:22.536193 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":194,"threads_started":1}
W20260619 02:25:22.536463 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:22.527417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32923 (local address 127.0.98.62:35451)
0619 02:25:22.527669 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:22.527672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:22.527762 (+ 90us) server_negotiation.cc:407] Connection header received
0619 02:25:22.527874 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:22.527876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:22.527918 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:22.528015 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:22.528420 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:22.528937 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:22.529833 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:22.530066 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:22.530214 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:22.536332 (+ 6118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32923: BlockingRecv error: recv got EOF from 127.0.98.1:32923 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":105,"threads_started":1}
Jun 19 02:25:22 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:22 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:22.715615 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:22.709590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:50333)
0619 02:25:22.709941 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:22.709965 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:22.710112 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:22.710455 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:22.710459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:22.710685 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:22.710793 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:22.710799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:22.711638 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:22.711643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:22.712286 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:22.712295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:22.712792 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:22.712796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:22.712898 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:22.713441 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:22.713453 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:22.715421 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":89,"threads_started":1}
W20260619 02:25:22.715678 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:22.709741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50333 (local address 127.0.98.62:35451)
0619 02:25:22.709905 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:22.709910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:22.710047 (+ 137us) server_negotiation.cc:407] Connection header received
0619 02:25:22.710224 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:22.710229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:22.710289 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:22.710433 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:22.710934 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:22.711516 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:22.712413 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:22.712669 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:22.712789 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:22.715560 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50333: BlockingRecv error: recv got EOF from 127.0.98.2:50333 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 02:25:22 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:22 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:22.914413 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:22.908484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:56641)
0619 02:25:22.908788 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:22.908803 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:22.908914 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:22.909164 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:22.909167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:22.909371 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:22.909485 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:22.909492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:22.910255 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:22.910257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:22.911092 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:22.911102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:22.911590 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:22.911594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:22.911682 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:22.912252 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:22.912266 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:22.914280 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":111,"threads_started":1}
W20260619 02:25:22.914557 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:22.908553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56641 (local address 127.0.98.62:35451)
0619 02:25:22.908721 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:22.908724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:22.908873 (+ 149us) server_negotiation.cc:407] Connection header received
0619 02:25:22.909000 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:22.909003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:22.909049 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:22.909120 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:22.909629 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:22.910154 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:22.911191 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:22.911450 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:22.911585 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:22.914405 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56641: BlockingRecv error: recv got EOF from 127.0.98.3:56641 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 02:25:23 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:23 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:23.542490 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:23.537184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:43479)
0619 02:25:23.537472 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:23.537486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:23.537648 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:23.537883 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:23.537885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:23.538063 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:23.538156 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:23.538161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:23.538950 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:23.538952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:23.539564 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:23.539573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:23.539972 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:23.539975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:23.540032 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:23.540502 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:23.540512 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:23.542366 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
W20260619 02:25:23.542650 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:23.537210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43479 (local address 127.0.98.62:35451)
0619 02:25:23.537522 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:23.537544 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:23.537588 (+ 44us) server_negotiation.cc:407] Connection header received
0619 02:25:23.537732 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:23.537734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:23.537777 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:23.537851 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:23.538298 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:23.538852 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:23.539661 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:23.539876 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:23.539952 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:23.542489 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43479: BlockingRecv error: recv got EOF from 127.0.98.1:43479 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":125,"threads_started":1}
Jun 19 02:25:23 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:23 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:23.721891 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:23.716606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:49081)
0619 02:25:23.716898 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:23.716913 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:23.717015 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:23.717243 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:23.717245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:23.717436 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:23.717534 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:23.717601 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:23.718301 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:23.718303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:23.718910 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:23.718918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:23.719328 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:23.719332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:23.719406 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:23.719894 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:23.719907 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:23.721768 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":106,"threads_started":1}
W20260619 02:25:23.722039 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:23.716610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49081 (local address 127.0.98.62:35451)
0619 02:25:23.716760 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:23.716764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:23.716983 (+ 219us) server_negotiation.cc:407] Connection header received
0619 02:25:23.717093 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:23.717096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:23.717137 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:23.717203 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:23.717707 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:23.718210 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:23.719008 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:23.719211 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:23.719286 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:23.721912 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49081: BlockingRecv error: recv got EOF from 127.0.98.2:49081 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 02:25:23 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:23 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:23.920346 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:23.915130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:33987)
0619 02:25:23.915377 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:23.915390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:23.915479 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:23.915694 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:23.915696 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:23.915913 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:23.916012 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:23.916017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:23.916830 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:23.916836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:23.917413 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:23.917422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:23.918002 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:23.918005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:23.918087 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:23.918544 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:23.918555 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:23.920232 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":84,"threads_started":1}
W20260619 02:25:23.920619 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:23.915330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33987 (local address 127.0.98.62:35451)
0619 02:25:23.915467 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:23.915471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:23.915484 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:25:23.915545 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:23.915547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:23.915587 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:23.915659 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:23.916121 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:23.916670 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:23.917534 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:23.917885 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:23.917997 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:23.920491 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33987: BlockingRecv error: recv got EOF from 127.0.98.3:33987 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 02:25:24 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:24 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:24.550701 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:24.543321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:45117)
0619 02:25:24.543620 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:24.543640 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:24.543751 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:24.544202 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:24.544206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:24.544478 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:24.544620 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:24.544630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:24.545628 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:24.545635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:24.546563 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:24.546576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:24.547177 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:24.547181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:24.547276 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:24.547896 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:24.547911 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:24.550486 (+ 2575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":99,"threads_started":1}
W20260619 02:25:24.550760 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:24.543543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45117 (local address 127.0.98.62:35451)
0619 02:25:24.543897 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:24.543904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:24.543921 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:25:24.543985 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:24.543988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:24.544049 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:24.544153 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:24.544756 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:24.545461 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:24.546699 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:24.547051 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:24.547153 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:24.550616 (+ 3463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45117: BlockingRecv error: recv got EOF from 127.0.98.1:45117 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":99,"threads_started":1}
Jun 19 02:25:24 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:24 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:24.728508 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:24.722819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:57717)
0619 02:25:24.723115 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:24.723129 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:24.723208 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:24.723453 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:24.723455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:24.723638 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:24.723732 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:24.723737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:24.724457 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:24.724460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:24.725036 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:24.725045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:24.725456 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:24.725460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:24.725528 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:24.726171 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:24.726183 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:24.728382 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":119,"threads_started":1}
W20260619 02:25:24.728770 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:24.723016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57717 (local address 127.0.98.62:35451)
0619 02:25:24.723164 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:24.723175 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:24.723191 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:25:24.723292 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:24.723295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:24.723347 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:24.723417 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:24.723837 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:24.724366 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:24.725134 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:24.725337 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:24.725409 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:24.728531 (+ 3122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57717: BlockingRecv error: recv got EOF from 127.0.98.2:57717 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 02:25:24 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:24 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:24.926774 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:24.921273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:55891)
0619 02:25:24.921560 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:24.921575 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:24.921688 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:24.922005 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:24.922008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:24.922221 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:24.922321 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:24.922327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:24.923079 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:24.923082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:24.923736 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:24.923744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:24.924184 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:24.924187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:24.924248 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:24.924836 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:24.924856 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:24.926645 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":100,"threads_started":1}
W20260619 02:25:24.926977 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:24.921281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55891 (local address 127.0.98.62:35451)
0619 02:25:24.921466 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:24.921471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:24.921691 (+ 220us) server_negotiation.cc:407] Connection header received
0619 02:25:24.921774 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:24.921779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:24.921862 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:24.921968 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:24.922423 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:24.922964 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:24.923864 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:24.924085 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:24.924166 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:24.926796 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55891: BlockingRecv error: recv got EOF from 127.0.98.3:55891 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 19 02:25:25 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:25 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:25.557796 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:25.551884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:33221)
0619 02:25:25.552162 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:25.552175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:25.552264 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:25.552602 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:25.552605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:25.552795 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:25.552893 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:25.552899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:25.553708 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:25.553711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:25.554544 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:25.554554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:25.555157 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:25.555160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:25.555277 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:25.555790 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:25.555802 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:25.557662 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":101,"threads_started":1}
W20260619 02:25:25.557991 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:25.552009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33221 (local address 127.0.98.62:35451)
0619 02:25:25.552236 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:25.552240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:25.552253 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:25:25.552348 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:25.552350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:25.552391 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:25.552468 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:25.553010 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:25.553602 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:25.554659 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:25.555024 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:25.555116 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:25.557810 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33221: BlockingRecv error: recv got EOF from 127.0.98.1:33221 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
Jun 19 02:25:25 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:25 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:25.735306 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:25.729279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:34891)
0619 02:25:25.729598 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:25.729611 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:25.729708 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:25.729985 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:25.729987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:25.730211 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:25.730319 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:25.730325 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:25.731057 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:25.731059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:25.731908 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:25.731917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:25.732574 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:25.732578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:25.732658 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:25.733210 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:25.733223 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:25.735132 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":100,"threads_started":1}
W20260619 02:25:25.735370 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:25.729352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34891 (local address 127.0.98.62:35451)
0619 02:25:25.729501 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:25.729507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:25.729684 (+ 177us) server_negotiation.cc:407] Connection header received
0619 02:25:25.729801 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:25.729803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:25.729846 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:25.729912 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:25.730425 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:25.730958 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:25.732063 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:25.732450 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:25.732561 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:25.735259 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34891: BlockingRecv error: recv got EOF from 127.0.98.2:34891 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 02:25:25 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:25 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:25.933950 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:25.927547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:54185)
0619 02:25:25.927884 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:25.927902 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:25.928050 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:25.928331 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:25.928334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:25.928552 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:25.928652 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:25.928658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:25.929522 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:25.929527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:25.930211 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:25.930222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:25.930813 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:25.930816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:25.930918 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:25.931598 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:25.931612 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:25.933779 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":83,"threads_started":1}
W20260619 02:25:25.934010 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:25.927605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54185 (local address 127.0.98.62:35451)
0619 02:25:25.927844 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:25.927855 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:25.927989 (+ 134us) server_negotiation.cc:407] Connection header received
0619 02:25:25.928162 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:25.928166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:25.928220 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:25.928306 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:25.928782 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:25.929407 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:25.930344 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:25.930658 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:25.930793 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:25.933895 (+ 3102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54185: BlockingRecv error: recv got EOF from 127.0.98.3:54185 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Jun 19 02:25:26 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:26 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:26.564162 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:26.558555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:47339)
0619 02:25:26.558894 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:26.558910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:26.559019 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:26.559284 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:26.559287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:26.559467 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:26.559598 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:26.559604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:26.560386 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:26.560388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:26.560940 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:26.560947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:26.561338 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:26.561340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:26.561403 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:26.562018 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:26.562042 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:26.564051 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":84,"threads_started":1}
W20260619 02:25:26.564303 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:26.558617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47339 (local address 127.0.98.62:35451)
0619 02:25:26.558896 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:26.558900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:26.559008 (+ 108us) server_negotiation.cc:407] Connection header received
0619 02:25:26.559099 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:26.559102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:26.559148 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:26.559229 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:26.559689 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:26.560292 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:26.561031 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:26.561244 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:26.561317 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:26.564174 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47339: BlockingRecv error: recv got EOF from 127.0.98.1:47339 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":70,"threads_started":1}
Jun 19 02:25:26 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:26 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:26.741727 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:26.736262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:48093)
0619 02:25:26.736569 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:26.736583 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:26.736683 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:26.736896 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:26.736897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:26.737129 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:26.737225 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:26.737231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:26.738137 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:26.738142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:26.738715 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:26.738725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:26.739208 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:26.739211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:26.739276 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:26.739757 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:26.739768 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:26.741612 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":87,"threads_started":1}
W20260619 02:25:26.741834 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:26.736302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48093 (local address 127.0.98.62:35451)
0619 02:25:26.736480 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:26.736484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:26.736650 (+ 166us) server_negotiation.cc:407] Connection header received
0619 02:25:26.736747 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:26.736749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:26.736790 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:26.736859 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:26.737331 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:26.737977 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:26.738827 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:26.739092 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:26.739171 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:26.741718 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48093: BlockingRecv error: recv got EOF from 127.0.98.2:48093 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 02:25:26 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:26 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:26.940879 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:26.934700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:54353)
0619 02:25:26.935070 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:26.935083 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:26.935238 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:26.935524 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:26.935527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:26.935739 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:26.935829 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:26.935834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:26.936587 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:26.936593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:26.937241 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:26.937250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:26.937811 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:26.937814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:26.937915 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:26.938533 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:26.938560 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:26.940695 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":106,"threads_started":1}
W20260619 02:25:26.940945 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:26.934798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54353 (local address 127.0.98.62:35451)
0619 02:25:26.934978 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:26.934983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:26.935166 (+ 183us) server_negotiation.cc:407] Connection header received
0619 02:25:26.935341 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:26.935345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:26.935407 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:26.935503 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:26.935962 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:26.936478 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:26.937351 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:26.937692 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:26.937799 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:26.940837 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54353: BlockingRecv error: recv got EOF from 127.0.98.3:54353 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 02:25:27 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:27 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:27.573186 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:27.564975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:51225)
0619 02:25:27.565259 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:27.565278 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:27.565363 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:27.565592 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:27.565594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:27.565810 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:27.565927 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:27.565932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:27.566686 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:27.566689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:27.567259 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:27.567266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:27.567665 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:27.567667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:27.567730 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:27.571209 (+ 3479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:27.571222 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:27.573019 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":114,"threads_started":1}
W20260619 02:25:27.573288 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:27.565123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51225 (local address 127.0.98.62:35451)
0619 02:25:27.565365 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:27.565368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:27.565380 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:25:27.565431 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:27.565433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:27.565472 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:27.565575 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:27.566058 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:27.566590 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:27.567356 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:27.567571 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:27.567645 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:27.573155 (+ 5510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51225: BlockingRecv error: recv got EOF from 127.0.98.1:51225 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":81,"threads_started":1}
Jun 19 02:25:27 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:27 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:27.748186 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:27.742556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:44789)
0619 02:25:27.742823 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:27.742838 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:27.742940 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:27.743143 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:27.743145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:27.743343 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:27.743442 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:27.743448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:27.744286 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:27.744288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:27.744892 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:27.744901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:27.745364 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:27.745366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:27.745465 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:27.746111 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:27.746124 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:27.748022 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":112,"threads_started":1}
W20260619 02:25:27.748241 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:27.742564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44789 (local address 127.0.98.62:35451)
0619 02:25:27.742716 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:27.742721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:27.742909 (+ 188us) server_negotiation.cc:407] Connection header received
0619 02:25:27.743027 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:27.743030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:27.743073 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:27.743129 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:27.743540 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:27.744193 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:27.745002 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:27.745270 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:27.745342 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:27.748136 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44789: BlockingRecv error: recv got EOF from 127.0.98.2:44789 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 02:25:27 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:27 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:27.947170 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:27.941695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:40673)
0619 02:25:27.942003 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:27.942017 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:27.942122 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:27.942356 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:27.942358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:27.942548 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:27.942643 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:27.942647 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:27.943341 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:27.943344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:27.943907 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:27.943914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:27.944302 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:27.944304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:27.944362 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:27.944933 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:27.944945 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:27.947001 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":112,"threads_started":1}
W20260619 02:25:27.947230 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:27.941736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40673 (local address 127.0.98.62:35451)
0619 02:25:27.941873 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:27.941878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:27.942090 (+ 212us) server_negotiation.cc:407] Connection header received
0619 02:25:27.942201 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:27.942203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:27.942251 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:27.942317 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:27.942736 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:27.943244 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:27.944001 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:27.944208 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:27.944283 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:27.947120 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40673: BlockingRecv error: recv got EOF from 127.0.98.3:40673 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 02:25:28 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:28 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:28.579787 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:28.573979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:37275)
0619 02:25:28.574264 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:28.574278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:28.574381 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:28.574684 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:28.574686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:28.574871 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:28.574967 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:28.574972 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:28.575981 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:28.575985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:28.576606 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:28.576618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:28.577113 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:28.577116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:28.577180 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:28.577782 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:28.577793 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:28.579672 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":90,"threads_started":1}
W20260619 02:25:28.579985 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:28.574026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37275 (local address 127.0.98.62:35451)
0619 02:25:28.574264 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:28.574269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:28.574351 (+ 82us) server_negotiation.cc:407] Connection header received
0619 02:25:28.574498 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:28.574500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:28.574548 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:28.574632 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:28.575093 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:28.575875 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:28.576718 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:28.577005 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:28.577091 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:28.579808 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37275: BlockingRecv error: recv got EOF from 127.0.98.1:37275 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":62,"threads_started":1}
Jun 19 02:25:28 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:28 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:28.754627 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:28.748973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:55743)
0619 02:25:28.749257 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:28.749271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:28.749374 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:28.749767 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:28.749769 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:28.749988 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:28.750089 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:28.750095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:28.750832 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:28.750836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:28.751495 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:28.751505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:28.751931 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:28.751934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:28.752011 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:28.752490 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:28.752503 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:28.754446 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":101,"threads_started":1}
W20260619 02:25:28.754688 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:28.749020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55743 (local address 127.0.98.62:35451)
0619 02:25:28.749147 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:28.749150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:28.749335 (+ 185us) server_negotiation.cc:407] Connection header received
0619 02:25:28.749452 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:28.749454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:28.749497 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:28.749618 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:28.750191 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:28.750711 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:28.751598 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:28.751807 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:28.751882 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:28.754582 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55743: BlockingRecv error: recv got EOF from 127.0.98.2:55743 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 02:25:28 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:28 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:28.954784 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:28.948028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:47939)
0619 02:25:28.948335 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:28.948349 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:28.948450 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:28.948718 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:28.948721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:28.948954 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:28.949058 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:28.949064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:28.949884 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:28.949887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:28.950510 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:28.950517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:28.951162 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:28.951167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:28.951268 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:28.952008 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:28.952024 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:28.954598 (+ 2574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":124,"threads_started":1}
W20260619 02:25:28.955251 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:28.948069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47939 (local address 127.0.98.62:35451)
0619 02:25:28.948219 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:28.948223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:28.948422 (+ 199us) server_negotiation.cc:407] Connection header received
0619 02:25:28.948528 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:28.948531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:28.948576 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:28.948644 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:28.949181 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:28.949769 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:28.950626 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:28.951027 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:28.951135 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:28.955117 (+ 3982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47939: BlockingRecv error: recv got EOF from 127.0.98.3:47939 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 02:25:29 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:29 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:29.586268 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:29.580567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:42217)
0619 02:25:29.580837 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:29.580852 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:29.580934 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:29.581215 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:29.581218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:29.581452 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:29.581621 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:29.581637 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:29.582529 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:29.582531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:29.583100 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:29.583107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:29.583591 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:29.583593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:29.583655 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:29.584154 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:29.584165 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:29.586107 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
W20260619 02:25:29.586400 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:29.580663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42217 (local address 127.0.98.62:35451)
0619 02:25:29.580889 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:29.580892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:29.580905 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:25:29.581022 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:29.581025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:29.581065 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:29.581145 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:29.581775 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:29.582310 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:29.583209 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:29.583488 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:29.583571 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:29.586260 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42217: BlockingRecv error: recv got EOF from 127.0.98.1:42217 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
Jun 19 02:25:29 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:29 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:29.761469 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:29.755573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:54011)
0619 02:25:29.755873 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:29.755886 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:29.755981 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:29.756243 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:29.756245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:29.756434 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:29.756529 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:29.756534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:29.757399 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:29.757403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:29.758191 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:29.758202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:29.758663 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:29.758667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:29.758749 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:29.759304 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:29.759316 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:29.761296 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":111,"threads_started":1}
W20260619 02:25:29.761596 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:29.755652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54011 (local address 127.0.98.62:35451)
0619 02:25:29.755812 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:29.755816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:29.755959 (+ 143us) server_negotiation.cc:407] Connection header received
0619 02:25:29.756074 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:29.756076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:29.756119 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:29.756199 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:29.756667 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:29.757249 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:29.758310 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:29.758545 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:29.758660 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:29.761428 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54011: BlockingRecv error: recv got EOF from 127.0.98.2:54011 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 02:25:29 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:29 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:29.961831 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:29.955593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:57749)
0619 02:25:29.955861 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:29.955878 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:29.956007 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:29.956333 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:29.956338 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:29.956561 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:29.956656 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:29.956662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:29.957784 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:29.957788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:29.958739 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:29.958754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:29.959308 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:29.959310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:29.959370 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:29.959879 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:29.959889 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:29.961711 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
W20260619 02:25:29.961957 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:29.955710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57749 (local address 127.0.98.62:35451)
0619 02:25:29.955856 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:29.955861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:29.955936 (+ 75us) server_negotiation.cc:407] Connection header received
0619 02:25:29.956105 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:29.956109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:29.956184 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:29.956261 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:29.956787 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:29.957682 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:29.958903 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:29.959207 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:29.959288 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:29.961839 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57749: BlockingRecv error: recv got EOF from 127.0.98.3:57749 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 02:25:30 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:30 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:30.592350 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:30.587060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:56961)
0619 02:25:30.587316 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:30.587332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:30.587432 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:30.587671 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:30.587673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:30.587842 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:30.587933 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:30.587937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:30.588702 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:30.588705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:30.589270 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:30.589277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:30.589709 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:30.589712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:30.589827 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:30.590333 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:30.590344 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:30.592236 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":66,"threads_started":1}
W20260619 02:25:30.592573 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:30.587131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56961 (local address 127.0.98.62:35451)
0619 02:25:30.587404 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:30.587408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:30.587421 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:25:30.587519 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:30.587521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:30.587562 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:30.587637 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:30.588051 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:30.588602 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:30.589369 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:30.589616 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:30.589694 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:30.592381 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56961: BlockingRecv error: recv got EOF from 127.0.98.1:56961 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
Jun 19 02:25:30 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:30 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:30.767565 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:30.762350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:41795)
0619 02:25:30.762640 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:30.762653 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:30.762746 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:30.762983 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:30.762986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:30.763164 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:30.763257 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:30.763261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:30.763979 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:30.763981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:30.764540 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:30.764547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:30.764986 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:30.764989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:30.765053 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:30.765590 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:30.765611 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:30.767437 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
W20260619 02:25:30.767731 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:30.762403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41795 (local address 127.0.98.62:35451)
0619 02:25:30.762549 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:30.762553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:30.762720 (+ 167us) server_negotiation.cc:407] Connection header received
0619 02:25:30.762828 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:30.762830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:30.762873 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:30.762946 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:30.763369 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:30.763881 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:30.764661 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:30.764877 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:30.764956 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:30.767573 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41795: BlockingRecv error: recv got EOF from 127.0.98.2:41795 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 02:25:30 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:30 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:30.968372 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:30.962647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:52053)
0619 02:25:30.963036 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:30.963049 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:30.963150 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:30.963360 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:30.963362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:30.963535 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:30.963627 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:30.963632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:30.964373 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:30.964375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:30.964930 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:30.964937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:30.965442 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:30.965446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:30.965532 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:30.966152 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:30.966169 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:30.968202 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":94,"threads_started":1}
W20260619 02:25:30.968426 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:30.962736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52053 (local address 127.0.98.62:35451)
0619 02:25:30.962880 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:30.962886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:30.963129 (+ 243us) server_negotiation.cc:407] Connection header received
0619 02:25:30.963219 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:30.963221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:30.963269 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:30.963347 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:30.963727 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:30.964277 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:30.965031 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:30.965248 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:30.965347 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:30.968319 (+ 2972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52053: BlockingRecv error: recv got EOF from 127.0.98.3:52053 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 02:25:31 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:31 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:31.598812 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:31.593050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:40511)
0619 02:25:31.593315 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:31.593329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:31.593415 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:31.593727 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:31.593730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:31.593941 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:31.594049 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:31.594055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:31.594785 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:31.594788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:31.595432 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:31.595441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:31.595952 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:31.595955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:31.596022 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:31.596610 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:31.596622 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:31.598641 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
W20260619 02:25:31.598869 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:31.593140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40511 (local address 127.0.98.62:35451)
0619 02:25:31.593360 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:31.593363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:31.593376 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:25:31.593497 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:31.593499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:31.593572 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:31.593648 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:31.594159 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:31.594685 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:31.595538 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:31.595774 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:31.595851 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:31.598753 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40511: BlockingRecv error: recv got EOF from 127.0.98.1:40511 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
Jun 19 02:25:31 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:31 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:31.773851 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:31.768403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:54915)
0619 02:25:31.768700 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:31.768714 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:31.768815 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:31.769037 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:31.769039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:31.769211 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:31.769301 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:31.769306 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:31.770067 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:31.770069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:31.770606 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:31.770613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:31.771029 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:31.771033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:31.771110 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:31.771743 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:31.771758 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:31.773739 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":97,"threads_started":1}
W20260619 02:25:31.774030 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:31.768463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54915 (local address 127.0.98.62:35451)
0619 02:25:31.768615 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:31.768621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:31.768786 (+ 165us) server_negotiation.cc:407] Connection header received
0619 02:25:31.768883 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:31.768885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:31.768929 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:31.768994 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:31.769394 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:31.769977 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:31.770699 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:31.770903 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:31.770976 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:31.773864 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54915: BlockingRecv error: recv got EOF from 127.0.98.2:54915 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 02:25:31 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:31 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:31.974154 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:31.969089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:40887)
0619 02:25:31.969337 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:31.969352 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:31.969462 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:31.969699 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:31.969701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:31.969876 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:31.969987 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:31.969992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:31.970688 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:31.970690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:31.971253 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:31.971260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:31.971637 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:31.971640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:31.971698 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:31.972161 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:31.972173 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:31.974032 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":91,"threads_started":1}
W20260619 02:25:31.974381 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:31.969175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40887 (local address 127.0.98.62:35451)
0619 02:25:31.969307 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:31.969311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:31.969407 (+ 96us) server_negotiation.cc:407] Connection header received
0619 02:25:31.969527 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:31.969529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:31.969592 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:31.969660 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:31.970090 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:31.970598 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:31.971346 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:31.971549 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:31.971620 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:31.974164 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40887: BlockingRecv error: recv got EOF from 127.0.98.3:40887 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 02:25:32 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:32 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:32.607904 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:32.599603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:58303)
0619 02:25:32.599862 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:32.599875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:32.599962 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:32.600200 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:32.600202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:32.600369 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:32.600461 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:32.600466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:32.601163 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:32.601166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:32.601740 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:32.601747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:32.602152 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:32.602154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:32.602213 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:32.605838 (+ 3625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:32.605850 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:32.607732 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
W20260619 02:25:32.607993 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:32.599683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58303 (local address 127.0.98.62:35451)
0619 02:25:32.599922 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:32.599926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:32.599939 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:25:32.600046 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:32.600049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:32.600091 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:32.600167 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:32.600551 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:32.601078 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:32.601834 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:32.602059 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:32.602135 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:32.607865 (+ 5730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58303: BlockingRecv error: recv got EOF from 127.0.98.1:58303 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":60,"threads_started":1}
Jun 19 02:25:32 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:32 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:32.780045 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:32.774673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:51433)
0619 02:25:32.774990 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:32.775005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:32.775135 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:32.775405 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:32.775408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:32.775631 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:32.775721 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:32.775727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:32.776477 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:32.776482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:32.777046 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:32.777054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:32.777521 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:32.777525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:32.777648 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:32.778124 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:32.778135 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:32.779922 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":111,"threads_started":1}
W20260619 02:25:32.780238 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:32.774816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51433 (local address 127.0.98.62:35451)
0619 02:25:32.774958 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:32.774962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:32.775091 (+ 129us) server_negotiation.cc:407] Connection header received
0619 02:25:32.775245 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:32.775248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:32.775300 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:32.775385 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:32.775835 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:32.776367 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:32.777165 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:32.777394 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:32.777521 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:32.780100 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51433: BlockingRecv error: recv got EOF from 127.0.98.2:51433 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 02:25:32 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:32 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:32.980434 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:32.974871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:37855)
0619 02:25:32.975140 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:32.975154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:32.975288 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:32.975571 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:32.975574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:32.975776 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:32.975868 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:32.975874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:32.976629 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:32.976633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:32.977198 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:32.977205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:32.977741 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:32.977744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:32.977854 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:32.978336 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:32.978347 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:32.980288 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
W20260619 02:25:32.980800 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:32.974950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37855 (local address 127.0.98.62:35451)
0619 02:25:32.975101 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:32.975106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:32.975232 (+ 126us) server_negotiation.cc:407] Connection header received
0619 02:25:32.975391 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:32.975394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:32.975451 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:32.975535 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:32.975978 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:32.976518 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:32.977321 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:32.977630 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:32.977736 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:32.980669 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37855: BlockingRecv error: recv got EOF from 127.0.98.3:37855 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 02:25:33 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:33 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:33.614580 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:33.608664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:54521)
0619 02:25:33.608933 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:33.608946 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:33.609033 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:33.609271 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:33.609273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:33.609442 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:33.609585 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:33.609591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:33.610414 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:33.610416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:33.610966 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:33.610973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:33.611720 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:33.611724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:33.611825 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:33.612307 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:33.612319 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:33.614358 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":103,"threads_started":1}
W20260619 02:25:33.614621 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:33.608734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54521 (local address 127.0.98.62:35451)
0619 02:25:33.609023 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:33.609027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:33.609039 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:25:33.609113 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:33.609115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:33.609157 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:33.609234 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:33.609698 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:33.610310 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:33.611110 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:33.611555 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:33.611706 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:33.614489 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54521: BlockingRecv error: recv got EOF from 127.0.98.1:54521 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":95,"threads_started":1}
Jun 19 02:25:33 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:33 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:33.786017 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:33.780760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:37553)
0619 02:25:33.781044 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:33.781057 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:33.781157 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:33.781365 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:33.781367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:33.781613 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:33.781706 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:33.781711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:33.782462 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:33.782465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:33.783041 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:33.783049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:33.783483 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:33.783485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:33.783544 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:33.784043 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:33.784055 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:33.785884 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":68,"threads_started":1}
W20260619 02:25:33.786149 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:33.780942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37553 (local address 127.0.98.62:35451)
0619 02:25:33.781067 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:33.781070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:33.781107 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:25:33.781221 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:33.781223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:33.781263 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:33.781327 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:33.781786 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:33.782319 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:33.783138 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:33.783358 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:33.783432 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:33.786008 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37553: BlockingRecv error: recv got EOF from 127.0.98.2:37553 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 02:25:33 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:33 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:33.986649 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:33.981181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.3:51487)
0619 02:25:33.981479 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:33.981495 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:33.981637 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:33.981913 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:33.981916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:33.982133 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:33.982224 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:33.982230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:33.982991 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:33.982997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:33.983576 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:33.983583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:33.984040 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:33.984043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:33.984126 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:33.984607 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:33.984619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:33.986480 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":85,"threads_started":1}
W20260619 02:25:33.986721 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:33.981261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51487 (local address 127.0.98.62:35451)
0619 02:25:33.981400 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:33.981404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:33.981569 (+ 165us) server_negotiation.cc:407] Connection header received
0619 02:25:33.981746 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:33.981750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:33.981806 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:33.981890 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:33.982340 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:33.982882 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:33.983689 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:33.983924 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:33.984036 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:33.986611 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51487: BlockingRecv error: recv got EOF from 127.0.98.3:51487 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 02:25:34 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:34 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:34.622495 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:34.615413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.1:42527)
0619 02:25:34.615685 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:34.615703 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:34.615791 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:34.616062 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:34.616064 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:34.616231 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:34.616320 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:34.616324 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:34.617404 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:34.617408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:34.618420 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:34.618434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:34.619113 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:34.619117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:34.619203 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:34.619774 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:34.619788 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:34.622300 (+ 2512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":65,"threads_started":1}
W20260619 02:25:34.622655 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:34.615490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42527 (local address 127.0.98.62:35451)
0619 02:25:34.615761 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:34.615765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:34.615778 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:25:34.615904 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:34.615906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:34.615947 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:34.616023 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:34.616430 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:34.617154 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:34.618572 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:34.619003 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:34.619111 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:34.622481 (+ 3370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42527: BlockingRecv error: recv got EOF from 127.0.98.1:42527 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
Jun 19 02:25:34 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:25:34 dist-test-slave-xkrt krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:25:34.792263 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:34.786856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35451 (local address 127.0.98.2:52509)
0619 02:25:34.787113 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:34.787126 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:34.787215 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:34.787424 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:34.787426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:34.787612 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:34.787703 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:34.787708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:34.788576 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:34.788580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:34.789255 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:34.789266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:34.789756 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:34.789759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:34.789844 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:34.790312 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:34.790323 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:34.792139 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
W20260619 02:25:34.792373 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:34.786911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52509 (local address 127.0.98.62:35451)
0619 02:25:34.787055 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:34.787058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:34.787190 (+ 132us) server_negotiation.cc:407] Connection header received
0619 02:25:34.787277 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:34.787280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:34.787320 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:34.787389 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:34.787795 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:34.788457 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:34.789379 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:34.789641 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:34.789750 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:34.792262 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52509: BlockingRecv error: recv got EOF from 127.0.98.2:52509 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:25:34.879518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 9120
I20260619 02:25:34.884809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 9272
I20260619 02:25:34.889962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 9405
I20260619 02:25:34.895020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 9050
2026-06-19T02:25:34Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17900 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T02:25:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:34Z Disabled control of system clock
I20260619 02:25:34.924842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38335
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 02:25:35.036801 9600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:35.037144 9600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:35.037230 9600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:35.041689 9600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:35.041764 9600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:35.041791 9600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:35.041828 9600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:35.041862 9600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:25:35.046710 9600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38335
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:35.047802 9600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:35.049165 9600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:35.057713 9605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:35.057741 9606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:35.057741 9608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:35.057792 9600 server_base.cc:1034] running on GCE node
I20260619 02:25:35.058452 9600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:35.059513 9600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:35.060680 9600 hybrid_clock.cc:648] HybridClock initialized: now 1781835935060669 us; error 25 us; skew 500 ppm
I20260619 02:25:35.062517 9600 webserver.cc:460] Webserver started at http://127.0.98.62:37433/ using document root <none> and password file <none>
I20260619 02:25:35.063088 9600 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:35.063149 9600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:35.063402 9600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:35.065372 9600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "cbe0c7cf6b3e46049528220c4ffc628f"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "877ce99fb9e3bcff4957e401da962692"
server_key_iv: "210f1b0a5d6b510c59cd11ca422e8022"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.065989 9600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "cbe0c7cf6b3e46049528220c4ffc628f"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "877ce99fb9e3bcff4957e401da962692"
server_key_iv: "210f1b0a5d6b510c59cd11ca422e8022"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.069470 9600 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:25:35.071834 9614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:35.073025 9600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:35.073128 9600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "cbe0c7cf6b3e46049528220c4ffc628f"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "877ce99fb9e3bcff4957e401da962692"
server_key_iv: "210f1b0a5d6b510c59cd11ca422e8022"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.073266 9600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:35.089103 9600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:35.089825 9600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:35.090025 9600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:35.096792 9600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38335
I20260619 02:25:35.096803 9666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38335 every 8 connection(s)
I20260619 02:25:35.097734 9600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:25:35.100293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 9600
I20260619 02:25:35.100404 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:25:35.100713 392 external_mini_cluster.cc:1428] Setting key ad56c3b593c996d5637dce2bf0bc0cb8
I20260619 02:25:35.101151 9667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:35.107177 9667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f: Bootstrap starting.
I20260619 02:25:35.109449 9667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:35.109905 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:35.102210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.0.98.62:38335)
0619 02:25:35.102812 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:35.102826 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:35.102860 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:25:35.103737 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:35.103755 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:35.104139 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:35.104479 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:35.104882 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:35.105872 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:35.106756 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:35.107213 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:35.107391 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:35.108223 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:35.108227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:35.108239 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:35.108599 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:35.108649 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:35.108668 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:35.108774 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:35.108914 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 02:25:35.109152 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":125,"threads_started":1}
I20260619 02:25:35.110306 9667 log.cc:824] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:35.112239 9667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f: No bootstrap required, opened a new log
I20260619 02:25:35.115435 9667 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe0c7cf6b3e46049528220c4ffc628f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38335 } }
I20260619 02:25:35.115640 9667 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:35.115679 9667 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe0c7cf6b3e46049528220c4ffc628f, State: Initialized, Role: FOLLOWER
I20260619 02:25:35.116016 9667 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe0c7cf6b3e46049528220c4ffc628f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38335 } }
I20260619 02:25:35.116119 9667 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:25:35.116154 9667 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:25:35.116219 9667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:35.116954 9667 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe0c7cf6b3e46049528220c4ffc628f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38335 } }
I20260619 02:25:35.117220 9667 leader_election.cc:302] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbe0c7cf6b3e46049528220c4ffc628f; no voters:
I20260619 02:25:35.117527 9667 leader_election.cc:288] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:25:35.117654 9672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:35.117867 9672 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [term 1 LEADER]: Becoming Leader. State: Replica: cbe0c7cf6b3e46049528220c4ffc628f, State: Running, Role: LEADER
I20260619 02:25:35.118186 9672 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe0c7cf6b3e46049528220c4ffc628f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38335 } }
I20260619 02:25:35.118722 9667 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:35.119724 9674 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbe0c7cf6b3e46049528220c4ffc628f. Latest consensus state: current_term: 1 leader_uuid: "cbe0c7cf6b3e46049528220c4ffc628f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe0c7cf6b3e46049528220c4ffc628f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38335 } } }
I20260619 02:25:35.119975 9673 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbe0c7cf6b3e46049528220c4ffc628f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe0c7cf6b3e46049528220c4ffc628f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38335 } } }
I20260619 02:25:35.120105 9674 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:35.120209 9673 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:35.120955 9681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:35.123557 9681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:35.128414 9681 catalog_manager.cc:1347] Generated new cluster ID: 46dd156fddea446f994d6a40f4d9f156
I20260619 02:25:35.128474 9681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:35.139850 9681 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:25:35.141001 9681 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:35.146039 9681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbe0c7cf6b3e46049528220c4ffc628f: Generated new TSK 0
I20260619 02:25:35.146749 9681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:25:35.155699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38335
--builtin_ntp_servers=127.0.98.20:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 02:25:35.264479 9691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:35.264716 9691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:35.264776 9691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:35.268743 9691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:35.268797 9691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:35.268893 9691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:25:35.273711 9691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:35.274691 9691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:35.275995 9691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:35.285394 9697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:35.285405 9699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:35.285406 9696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:35.285745 9691 server_base.cc:1034] running on GCE node
I20260619 02:25:35.286123 9691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:35.286778 9691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:35.287961 9691 hybrid_clock.cc:648] HybridClock initialized: now 1781835935287949 us; error 34 us; skew 500 ppm
I20260619 02:25:35.289718 9691 webserver.cc:460] Webserver started at http://127.0.98.1:45539/ using document root <none> and password file <none>
I20260619 02:25:35.290271 9691 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:35.290364 9691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:35.290566 9691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:35.292490 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "81a2f4c953bd4905b2d3aadbd9204fff"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "3ddd4be20255853d6d5d51c641bf7a35"
server_key_iv: "f4c434b2f3f9ab4d3a86336071209b9a"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.293049 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "81a2f4c953bd4905b2d3aadbd9204fff"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "3ddd4be20255853d6d5d51c641bf7a35"
server_key_iv: "f4c434b2f3f9ab4d3a86336071209b9a"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.296433 9691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:25:35.298799 9705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:35.300053 9691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:25:35.300181 9691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "81a2f4c953bd4905b2d3aadbd9204fff"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "3ddd4be20255853d6d5d51c641bf7a35"
server_key_iv: "f4c434b2f3f9ab4d3a86336071209b9a"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.300285 9691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:35.323947 9691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:35.324602 9691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:35.324795 9691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:35.325342 9691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:35.326285 9691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:35.326359 9691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:35.326426 9691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:35.326473 9691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:35.334929 9691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45287
I20260619 02:25:35.334954 9818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45287 every 8 connection(s)
I20260619 02:25:35.335779 9691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:25:35.341706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 9691
I20260619 02:25:35.341831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:25:35.342113 392 external_mini_cluster.cc:1428] Setting key 17f761c8287faf1747777bec6b95501f
I20260619 02:25:35.344270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38335
--builtin_ntp_servers=127.0.98.20:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 02:25:35.345014 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:35.337525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41675 (local address 127.0.98.62:38335)
0619 02:25:35.337708 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:35.337719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:35.338641 (+ 922us) server_negotiation.cc:407] Connection header received
0619 02:25:35.339937 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:35.339939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:35.339984 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:35.340064 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:35.340892 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:35.341469 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:35.342849 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:35.343138 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:35.343245 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:35.344402 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:35.344408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:35.344413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:35.344429 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:35.344455 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:35.344469 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:35.344553 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:35.344719 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 02:25:35.344794 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 02:25:35.345741 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:35.337917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38335 (local address 127.0.98.1:41675)
0619 02:25:35.338518 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:35.338556 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:35.339751 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:35.340232 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:35.340238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:35.340318 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:35.340774 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:35.340785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:35.341611 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:35.341615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:35.342720 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:35.342731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:35.343301 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:35.343308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:35.343495 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:35.344137 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:25:35.344138 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:35.344176 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:25:35.344236 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:35.344555 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:35.344563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:35.344568 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:25:35.344665 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 02:25:35.344872 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":97,"threads_started":1}
I20260619 02:25:35.346910 9819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38335
I20260619 02:25:35.347150 9819 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:35.347635 9819 heartbeater.cc:511] Master 127.0.98.62:38335 requested a full tablet report, sending...
I20260619 02:25:35.349253 9631 ts_manager.cc:194] Registered new tserver with Master: 81a2f4c953bd4905b2d3aadbd9204fff (127.0.98.1:45287)
I20260619 02:25:35.351016 9631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41675
W20260619 02:25:35.470969 9822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:35.471206 9822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:35.471242 9822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:35.475214 9822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:35.475261 9822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:35.475318 9822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:25:35.479974 9822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:35.480902 9822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:35.482242 9822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:35.491290 9827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:35.491353 9828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:35.491465 9822 server_base.cc:1034] running on GCE node
W20260619 02:25:35.491729 9830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:35.492117 9822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:35.492697 9822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:35.493851 9822 hybrid_clock.cc:648] HybridClock initialized: now 1781835935493839 us; error 23 us; skew 500 ppm
I20260619 02:25:35.495486 9822 webserver.cc:460] Webserver started at http://127.0.98.2:39941/ using document root <none> and password file <none>
I20260619 02:25:35.496061 9822 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:35.496120 9822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:35.496349 9822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:35.498301 9822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "a0358df3a1ae4fabb135621a65d2ab32"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "775c89345a9f42add90b4209bb057495"
server_key_iv: "6532828d5e19a13849f9b28ee75ca6c7"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.498848 9822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a0358df3a1ae4fabb135621a65d2ab32"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "775c89345a9f42add90b4209bb057495"
server_key_iv: "6532828d5e19a13849f9b28ee75ca6c7"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.502364 9822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:25:35.504735 9836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:35.505945 9822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:35.506085 9822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "a0358df3a1ae4fabb135621a65d2ab32"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "775c89345a9f42add90b4209bb057495"
server_key_iv: "6532828d5e19a13849f9b28ee75ca6c7"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.506196 9822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:35.528436 9822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:35.529096 9822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:35.529282 9822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:35.529906 9822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:35.530813 9822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:35.530885 9822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:35.530956 9822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:35.530998 9822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:35.539578 9822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39185
I20260619 02:25:35.539614 9949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39185 every 8 connection(s)
I20260619 02:25:35.540437 9822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:25:35.540844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 9822
I20260619 02:25:35.540969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:25:35.541318 392 external_mini_cluster.cc:1428] Setting key 5d76a31e70b56887f3216823912f5ebf
I20260619 02:25:35.543303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38335
--builtin_ntp_servers=127.0.98.20:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 02:25:35.550630 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:35.542327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41245 (local address 127.0.98.62:38335)
0619 02:25:35.542468 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:35.542473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:35.543659 (+ 1186us) server_negotiation.cc:407] Connection header received
0619 02:25:35.545290 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:35.545293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:35.545337 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:35.545401 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:35.546427 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:35.547323 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:35.548404 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:35.548882 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:35.549020 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:35.550079 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:35.550085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:35.550088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:35.550105 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:35.550132 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:35.550145 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:35.550234 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:35.550363 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 02:25:35.550435 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:25:35.551317 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:35.542803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38335 (local address 127.0.98.2:41245)
0619 02:25:35.543506 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:35.543557 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:35.545050 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:35.545633 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:35.545645 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:35.545756 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:35.546286 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:35.546298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:35.547436 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:35.547438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:35.548285 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:35.548293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:35.549019 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:35.549021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:35.549168 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:35.549822 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:25:35.549823 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:35.549861 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:25:35.549921 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:35.550231 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:35.550237 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:35.550241 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:25:35.550329 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 02:25:35.550527 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":123,"threads_started":1}
I20260619 02:25:35.552400 9950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38335
I20260619 02:25:35.552623 9950 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:35.553089 9950 heartbeater.cc:511] Master 127.0.98.62:38335 requested a full tablet report, sending...
I20260619 02:25:35.554091 9631 ts_manager.cc:194] Registered new tserver with Master: a0358df3a1ae4fabb135621a65d2ab32 (127.0.98.2:39185)
I20260619 02:25:35.554793 9631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41245
W20260619 02:25:35.653959 9953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:35.654204 9953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:35.654239 9953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:35.658257 9953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:35.658308 9953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:35.658360 9953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:25:35.662969 9953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:35.663961 9953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:35.665206 9953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:35.674368 9959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:35.674381 9958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:35.674566 9953 server_base.cc:1034] running on GCE node
W20260619 02:25:35.674377 9961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:35.675058 9953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:35.675701 9953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:35.676882 9953 hybrid_clock.cc:648] HybridClock initialized: now 1781835935676856 us; error 46 us; skew 500 ppm
I20260619 02:25:35.678979 9953 webserver.cc:460] Webserver started at http://127.0.98.3:33621/ using document root <none> and password file <none>
I20260619 02:25:35.679526 9953 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:35.679592 9953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:35.679852 9953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:35.681857 9953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "264e2d00eba04dae90e696c640764d7d"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "aabba944ecfa757bae2cbc5dced8773c"
server_key_iv: "5537437ac70b6109ae0bcbff394709a0"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.682484 9953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "264e2d00eba04dae90e696c640764d7d"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "aabba944ecfa757bae2cbc5dced8773c"
server_key_iv: "5537437ac70b6109ae0bcbff394709a0"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.685927 9953 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:25:35.688310 9967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:35.689500 9953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:35.689666 9953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "264e2d00eba04dae90e696c640764d7d"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "aabba944ecfa757bae2cbc5dced8773c"
server_key_iv: "5537437ac70b6109ae0bcbff394709a0"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.689776 9953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:35.709229 9953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:35.710007 9953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:35.710197 9953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:35.710772 9953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:35.711655 9953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:35.711723 9953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:35.711784 9953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:35.711834 9953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:35.720468 9953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39677
I20260619 02:25:35.720495 10080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39677 every 8 connection(s)
I20260619 02:25:35.721357 9953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:25:35.729599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 9953
I20260619 02:25:35.729734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:25:35.729892 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:35.723099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56341 (local address 127.0.98.62:38335)
0619 02:25:35.723227 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:35.723231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:35.724233 (+ 1002us) server_negotiation.cc:407] Connection header received
0619 02:25:35.725517 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:35.725520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:35.725617 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:35.725702 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:35.726502 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:35.727004 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:35.728048 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:35.728247 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:35.728316 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:35.729362 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:35.729367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:35.729370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:35.729385 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:35.729403 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:35.729414 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:35.729494 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:35.729688 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 02:25:35.729749 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:25:35.730029 392 external_mini_cluster.cc:1428] Setting key 8091836ec6d05f5184069677e4f25d16
I20260619 02:25:35.730695 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:35.723487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38335 (local address 127.0.98.3:56341)
0619 02:25:35.724088 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:35.724126 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:35.725343 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:35.725854 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:35.725861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:35.725938 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:35.726387 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:35.726398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:35.727109 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:35.727111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:35.727946 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:35.727954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:35.728357 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:35.728360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:35.728493 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:35.729096 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:25:35.729098 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:35.729136 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:25:35.729193 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:35.729499 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:35.729504 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:35.729508 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:25:35.729660 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 02:25:35.729922 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":101,"threads_started":1}
I20260619 02:25:35.731689 10081 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38335
I20260619 02:25:35.731911 10081 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:35.732405 10081 heartbeater.cc:511] Master 127.0.98.62:38335 requested a full tablet report, sending...
I20260619 02:25:35.733273 9631 ts_manager.cc:194] Registered new tserver with Master: 264e2d00eba04dae90e696c640764d7d (127.0.98.3:39677)
I20260619 02:25:35.733881 9631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56341
I20260619 02:25:35.743466 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:25:35.746055 9670 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:35.745185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.0.98.62:38335)
0619 02:25:35.745333 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:35.745339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:35.745450 (+ 111us) server_negotiation.cc:407] Connection header received
0619 02:25:35.745592 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:35.745594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:35.745631 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:35.745693 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:35.745912 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46794: BlockingRecv error: recv got EOF from 127.0.0.1:46794 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:25:35.746155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 9691
I20260619 02:25:35.751856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 9822
I20260619 02:25:35.756902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 9953
I20260619 02:25:35.761821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 9600
2026-06-19T02:25:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (864 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T02:25:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:35Z Disabled control of system clock
I20260619 02:25:35.789451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45599
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:25:35.896194 10092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:35.896454 10092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:35.896490 10092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:35.900347 10092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:35.900450 10092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:35.900477 10092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:35.900499 10092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:35.900522 10092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:25:35.905297 10092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:35.906409 10092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:35.907747 10092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:35.916014 10097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:35.916124 10098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:35.916154 10092 server_base.cc:1034] running on GCE node
W20260619 02:25:35.916317 10100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:35.916832 10092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:35.917996 10092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:35.919157 10092 hybrid_clock.cc:648] HybridClock initialized: now 1781835935919145 us; error 25 us; skew 500 ppm
I20260619 02:25:35.920984 10092 webserver.cc:460] Webserver started at http://127.0.98.62:41645/ using document root <none> and password file <none>
I20260619 02:25:35.921610 10092 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:35.921677 10092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:35.921917 10092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:35.923910 10092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "804526b93a664c33ab6f84b798cc2b5a"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "1617aaf5deb034554a65d3c8548bbce8"
server_key_iv: "fffd25dafb6049ba479145930eaa263d"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.924494 10092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "804526b93a664c33ab6f84b798cc2b5a"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "1617aaf5deb034554a65d3c8548bbce8"
server_key_iv: "fffd25dafb6049ba479145930eaa263d"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.928074 10092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:25:35.930538 10106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:35.931761 10092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:35.931885 10092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "804526b93a664c33ab6f84b798cc2b5a"
format_stamp: "Formatted at 2026-06-19 02:25:35 on dist-test-slave-xkrt"
server_key: "1617aaf5deb034554a65d3c8548bbce8"
server_key_iv: "fffd25dafb6049ba479145930eaa263d"
server_key_version: "encryptionkey@0"
I20260619 02:25:35.931988 10092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:35.936520 10092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:35.937105 10092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:35.937282 10092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:35.943892 10092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45599
I20260619 02:25:35.943914 10158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45599 every 8 connection(s)
I20260619 02:25:35.944798 10092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:25:35.947980 10159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:35.953204 10159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a: Bootstrap starting.
I20260619 02:25:35.954622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 10092
I20260619 02:25:35.954748 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:25:35.955014 392 external_mini_cluster.cc:1428] Setting key 3c3d80dff49a1e7f604ff9e27ea196c2
I20260619 02:25:35.955806 10159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:35.956557 10159 log.cc:824] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:35.958601 10159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a: No bootstrap required, opened a new log
I20260619 02:25:35.960180 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:35.956125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.0.98.62:45599)
0619 02:25:35.956812 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:35.956828 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:35.956861 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:25:35.957471 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:35.957488 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:35.957847 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:35.958123 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:35.958944 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:35.958948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:35.958959 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:35.958969 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:35.959020 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:35.959037 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:35.959128 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:35.959263 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 02:25:35.959522 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":157,"threads_started":1}
I20260619 02:25:35.962486 10159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804526b93a664c33ab6f84b798cc2b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45599 } }
I20260619 02:25:35.962719 10159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:35.962795 10159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804526b93a664c33ab6f84b798cc2b5a, State: Initialized, Role: FOLLOWER
I20260619 02:25:35.963238 10159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804526b93a664c33ab6f84b798cc2b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45599 } }
I20260619 02:25:35.963397 10159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:25:35.963482 10159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:25:35.963577 10159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:35.964454 10159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804526b93a664c33ab6f84b798cc2b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45599 } }
I20260619 02:25:35.964785 10159 leader_election.cc:302] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 804526b93a664c33ab6f84b798cc2b5a; no voters:
I20260619 02:25:35.965076 10159 leader_election.cc:288] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:25:35.965185 10164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:35.965423 10164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [term 1 LEADER]: Becoming Leader. State: Replica: 804526b93a664c33ab6f84b798cc2b5a, State: Running, Role: LEADER
I20260619 02:25:35.965778 10164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804526b93a664c33ab6f84b798cc2b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45599 } }
I20260619 02:25:35.966377 10159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:35.967525 10165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "804526b93a664c33ab6f84b798cc2b5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804526b93a664c33ab6f84b798cc2b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45599 } } }
I20260619 02:25:35.967556 10166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 804526b93a664c33ab6f84b798cc2b5a. Latest consensus state: current_term: 1 leader_uuid: "804526b93a664c33ab6f84b798cc2b5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804526b93a664c33ab6f84b798cc2b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45599 } } }
I20260619 02:25:35.967659 10166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:35.967659 10165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:35.968087 10173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:35.971385 10173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:35.976169 10173 catalog_manager.cc:1347] Generated new cluster ID: e79cd3fee12c476d9cd119984c579f8e
I20260619 02:25:35.976231 10173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:35.995771 10173 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:25:35.996109 10173 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:35.999958 10173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 804526b93a664c33ab6f84b798cc2b5a: Generated new TSK 0
I20260619 02:25:36.000639 10173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:25:36.010779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45599
--builtin_ntp_servers=127.0.98.20:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:25:36.137269 10183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:36.137511 10183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:36.137569 10183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:36.141662 10183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:36.141708 10183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:36.141789 10183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:25:36.146391 10183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:36.147382 10183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:36.148821 10183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:36.158326 10188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:36.158375 10189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:36.158337 10191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:36.158603 10183 server_base.cc:1034] running on GCE node
I20260619 02:25:36.159006 10183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:36.159564 10183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:36.160772 10183 hybrid_clock.cc:648] HybridClock initialized: now 1781835936160745 us; error 52 us; skew 500 ppm
I20260619 02:25:36.162535 10183 webserver.cc:460] Webserver started at http://127.0.98.1:41109/ using document root <none> and password file <none>
I20260619 02:25:36.163084 10183 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:36.163173 10183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:36.163401 10183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:36.165351 10183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "cfca0b828ca04ebea8b289c2b5dcc28b"
format_stamp: "Formatted at 2026-06-19 02:25:36 on dist-test-slave-xkrt"
server_key: "6bd074a35b47cf175733c68b28de9bc5"
server_key_iv: "c1b8ba50b9a815ade2c149924fe35f69"
server_key_version: "encryptionkey@0"
I20260619 02:25:36.166018 10183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cfca0b828ca04ebea8b289c2b5dcc28b"
format_stamp: "Formatted at 2026-06-19 02:25:36 on dist-test-slave-xkrt"
server_key: "6bd074a35b47cf175733c68b28de9bc5"
server_key_iv: "c1b8ba50b9a815ade2c149924fe35f69"
server_key_version: "encryptionkey@0"
I20260619 02:25:36.170135 10183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:25:36.172573 10197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:36.173837 10183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:36.173966 10183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "cfca0b828ca04ebea8b289c2b5dcc28b"
format_stamp: "Formatted at 2026-06-19 02:25:36 on dist-test-slave-xkrt"
server_key: "6bd074a35b47cf175733c68b28de9bc5"
server_key_iv: "c1b8ba50b9a815ade2c149924fe35f69"
server_key_version: "encryptionkey@0"
I20260619 02:25:36.174079 10183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:36.178611 10183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:36.179174 10183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:36.179349 10183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:36.179867 10183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:36.180816 10183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:36.180884 10183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:36.180939 10183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:36.180979 10183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:36.190740 10183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43195
I20260619 02:25:36.190757 10310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43195 every 8 connection(s)
I20260619 02:25:36.191614 10183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:25:36.197397 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:36.193306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49979 (local address 127.0.98.62:45599)
0619 02:25:36.193447 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:36.193451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:36.194497 (+ 1046us) server_negotiation.cc:407] Connection header received
0619 02:25:36.195789 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:36.195792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:36.195835 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:36.195917 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:36.196926 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:36.196929 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:36.196932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:36.196944 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:36.196964 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:36.196980 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:36.197081 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:36.197222 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 02:25:36.197266 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:25:36.197577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 10183
I20260619 02:25:36.197692 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:25:36.197985 392 external_mini_cluster.cc:1428] Setting key 41fa5e89716de53d7d19eca102f4b1ef
I20260619 02:25:36.198354 10313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:36.193721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45599 (local address 127.0.98.1:49979)
0619 02:25:36.194342 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:36.194381 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:36.195586 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:36.196077 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:36.196083 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:36.196166 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:36.196687 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:25:36.196688 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:36.196726 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:25:36.196787 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:36.197068 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:36.197076 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:36.197081 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:25:36.197251 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 02:25:36.197472 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20260619 02:25:36.199589 10311 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45599
I20260619 02:25:36.199810 10311 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:36.199991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45599
--builtin_ntp_servers=127.0.98.20:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 02:25:36.200249 10311 heartbeater.cc:511] Master 127.0.98.62:45599 requested a full tablet report, sending...
I20260619 02:25:36.201795 10123 ts_manager.cc:194] Registered new tserver with Master: cfca0b828ca04ebea8b289c2b5dcc28b (127.0.98.1:43195)
W20260619 02:25:36.306419 10314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:36.306733 10314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:36.306771 10314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:36.310650 10314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:36.310699 10314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:36.310752 10314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:25:36.315230 10314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:36.316144 10314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:36.317368 10314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:36.326400 10320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:36.326376 10319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:36.326586 10314 server_base.cc:1034] running on GCE node
W20260619 02:25:36.326598 10322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:36.327039 10314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:36.327649 10314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:36.328826 10314 hybrid_clock.cc:648] HybridClock initialized: now 1781835936328799 us; error 46 us; skew 500 ppm
I20260619 02:25:36.330554 10314 webserver.cc:460] Webserver started at http://127.0.98.2:34629/ using document root <none> and password file <none>
I20260619 02:25:36.331091 10314 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:36.331176 10314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:36.331395 10314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:36.333297 10314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "ebac663a97f040ed9537c26ac7f6b6e0"
format_stamp: "Formatted at 2026-06-19 02:25:36 on dist-test-slave-xkrt"
server_key: "96e1ff9635f81204beefc8d8fb64aa04"
server_key_iv: "7dce3fa546751f909d4e7722425ffdf0"
server_key_version: "encryptionkey@0"
I20260619 02:25:36.333864 10314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ebac663a97f040ed9537c26ac7f6b6e0"
format_stamp: "Formatted at 2026-06-19 02:25:36 on dist-test-slave-xkrt"
server_key: "96e1ff9635f81204beefc8d8fb64aa04"
server_key_iv: "7dce3fa546751f909d4e7722425ffdf0"
server_key_version: "encryptionkey@0"
I20260619 02:25:36.337280 10314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 02:25:36.339496 10328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:36.340627 10314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:36.340752 10314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "ebac663a97f040ed9537c26ac7f6b6e0"
format_stamp: "Formatted at 2026-06-19 02:25:36 on dist-test-slave-xkrt"
server_key: "96e1ff9635f81204beefc8d8fb64aa04"
server_key_iv: "7dce3fa546751f909d4e7722425ffdf0"
server_key_version: "encryptionkey@0"
I20260619 02:25:36.340855 10314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:36.345232 10314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:36.345835 10314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:36.346011 10314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:36.346506 10314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:36.347333 10314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:36.347410 10314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:36.347714 10314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:36.347782 10314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:36.356349 10314 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35575
I20260619 02:25:36.356374 10441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35575 every 8 connection(s)
I20260619 02:25:36.357180 10314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:25:36.362761 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:36.358869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32887 (local address 127.0.98.62:45599)
0619 02:25:36.359000 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:36.359003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:36.359977 (+ 974us) server_negotiation.cc:407] Connection header received
0619 02:25:36.361200 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:36.361203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:36.361239 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:36.361315 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:36.362401 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:36.362403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:36.362405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:36.362415 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:36.362433 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:36.362440 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:36.362513 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:36.362621 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 02:25:36.362656 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:25:36.363603 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:36.359251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45599 (local address 127.0.98.2:32887)
0619 02:25:36.359857 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:36.359894 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:36.361008 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:36.361462 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:36.361469 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:36.361596 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:36.362180 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:25:36.362182 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:36.362219 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:25:36.362280 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:36.362510 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:36.362515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:36.362520 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:25:36.362642 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 02:25:36.362895 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20260619 02:25:36.364670 10442 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45599
I20260619 02:25:36.364889 10442 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:36.365260 10442 heartbeater.cc:511] Master 127.0.98.62:45599 requested a full tablet report, sending...
I20260619 02:25:36.365823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 10314
I20260619 02:25:36.365964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:25:36.366320 392 external_mini_cluster.cc:1428] Setting key bccbd5bc1fd2382e94c5e2f2d14e802e
I20260619 02:25:36.366432 10123 ts_manager.cc:194] Registered new tserver with Master: ebac663a97f040ed9537c26ac7f6b6e0 (127.0.98.2:35575)
I20260619 02:25:36.368330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45599
--builtin_ntp_servers=127.0.98.20:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:25:36.475485 10445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:36.475716 10445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:36.475754 10445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:36.479622 10445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:36.479671 10445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:36.479723 10445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:25:36.484840 10445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:36.485855 10445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:36.487123 10445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:25:36.496418 10445 server_base.cc:1034] running on GCE node
W20260619 02:25:36.496381 10453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:36.496368 10450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:36.496364 10451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:36.496974 10445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:36.497610 10445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:36.498772 10445 hybrid_clock.cc:648] HybridClock initialized: now 1781835936498759 us; error 25 us; skew 500 ppm
I20260619 02:25:36.500470 10445 webserver.cc:460] Webserver started at http://127.0.98.3:43785/ using document root <none> and password file <none>
I20260619 02:25:36.500996 10445 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:36.501055 10445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:36.501214 10445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:36.503324 10445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "e6f29a4225284653a2f45b7d4029a899"
format_stamp: "Formatted at 2026-06-19 02:25:36 on dist-test-slave-xkrt"
server_key: "8e0d9ba2db8b3fef6b45272634ec4d4e"
server_key_iv: "fafa58f4a91a2c258be4ded4137779d3"
server_key_version: "encryptionkey@0"
I20260619 02:25:36.503893 10445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e6f29a4225284653a2f45b7d4029a899"
format_stamp: "Formatted at 2026-06-19 02:25:36 on dist-test-slave-xkrt"
server_key: "8e0d9ba2db8b3fef6b45272634ec4d4e"
server_key_iv: "fafa58f4a91a2c258be4ded4137779d3"
server_key_version: "encryptionkey@0"
I20260619 02:25:36.507387 10445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:25:36.509795 10459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:36.510947 10445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:25:36.511071 10445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "e6f29a4225284653a2f45b7d4029a899"
format_stamp: "Formatted at 2026-06-19 02:25:36 on dist-test-slave-xkrt"
server_key: "8e0d9ba2db8b3fef6b45272634ec4d4e"
server_key_iv: "fafa58f4a91a2c258be4ded4137779d3"
server_key_version: "encryptionkey@0"
I20260619 02:25:36.511174 10445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:36.515650 10445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:36.516201 10445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:36.516378 10445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:36.516898 10445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:36.517812 10445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:36.517859 10445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:36.517921 10445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:36.517961 10445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:36.526907 10445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42405
I20260619 02:25:36.526932 10572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42405 every 8 connection(s)
I20260619 02:25:36.527788 10445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:25:36.533391 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:36.529498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39075 (local address 127.0.98.62:45599)
0619 02:25:36.529677 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:36.529681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:36.530683 (+ 1002us) server_negotiation.cc:407] Connection header received
0619 02:25:36.531922 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:36.531924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:36.531960 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:36.532036 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:36.533020 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:36.533023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:36.533025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:25:36.533035 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:36.533052 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:25:36.533060 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:36.533132 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:36.533252 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 02:25:36.533288 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 02:25:36.533787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 10445
I20260619 02:25:36.533900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:25:36.534209 392 external_mini_cluster.cc:1428] Setting key a427b188f1a115c5416f0d0c1ec66764
I20260619 02:25:36.534886 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:36.529903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45599 (local address 127.0.98.3:39075)
0619 02:25:36.530526 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:36.530564 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:36.531742 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:36.532196 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:36.532202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:36.532285 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:36.532784 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:25:36.532786 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:36.532822 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:25:36.532886 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:36.533135 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:36.533141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:36.533145 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:25:36.533265 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 02:25:36.533438 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":118,"threads_started":1}
I20260619 02:25:36.536026 10573 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45599
I20260619 02:25:36.536273 10573 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:36.536717 10573 heartbeater.cc:511] Master 127.0.98.62:45599 requested a full tablet report, sending...
I20260619 02:25:36.537700 10122 ts_manager.cc:194] Registered new tserver with Master: e6f29a4225284653a2f45b7d4029a899 (127.0.98.3:42405)
I20260619 02:25:36.547940 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:25:36.550566 10162 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:25:36.549821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.0.98.62:45599)
0619 02:25:36.549936 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:36.549941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:36.550143 (+ 202us) server_negotiation.cc:407] Connection header received
0619 02:25:36.550231 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:36.550234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:36.550262 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:36.550321 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:36.550469 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37514: BlockingRecv error: recv got EOF from 127.0.0.1:37514 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:25:36.550866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 10183
I20260619 02:25:36.556407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 10314
I20260619 02:25:36.561955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 10445
I20260619 02:25:36.567376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 10092
2026-06-19T02:25:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (806 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:25:36 dist-test-slave-xkrt krb5kdc[10583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:25:36 dist-test-slave-xkrt krb5kdc[10583](info): set up 2 sockets
Jun 19 02:25:36 dist-test-slave-xkrt krb5kdc[10583](info): commencing operation
krb5kdc: starting...
W20260619 02:25:37.598507 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:25:37 dist-test-slave-xkrt krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:25:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:25:37.714485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44089
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:37.820422 10596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:37.820669 10596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:37.820708 10596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:37.824636 10596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:25:37.824692 10596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:37.824712 10596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:37.824728 10596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:25:37.824743 10596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:25:37.829895 10596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:37.830837 10596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:37.831743 10596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:37.840199 10601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:37.840236 10602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:37.840230 10604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:37.840304 10596 server_base.cc:1034] running on GCE node
I20260619 02:25:37.840905 10596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:37.842022 10596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:37.843214 10596 hybrid_clock.cc:648] HybridClock initialized: now 1781835937843202 us; error 29 us; skew 500 ppm
Jun 19 02:25:37 dist-test-slave-xkrt krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:37.848815 10596 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:25:37.849795 10596 webserver.cc:460] Webserver started at http://127.0.98.62:32905/ using document root <none> and password file <none>
I20260619 02:25:37.850360 10596 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:37.850452 10596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:37.850682 10596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:37.852689 10596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "953452fb0a20456ba1dbb9a0112db9b9"
format_stamp: "Formatted at 2026-06-19 02:25:37 on dist-test-slave-xkrt"
server_key: "bc375a0e659346d8f10314725b5fe761"
server_key_iv: "2642b2a9abedc7efdb8fdd5fb2895323"
server_key_version: "encryptionkey@0"
I20260619 02:25:37.853251 10596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "953452fb0a20456ba1dbb9a0112db9b9"
format_stamp: "Formatted at 2026-06-19 02:25:37 on dist-test-slave-xkrt"
server_key: "bc375a0e659346d8f10314725b5fe761"
server_key_iv: "2642b2a9abedc7efdb8fdd5fb2895323"
server_key_version: "encryptionkey@0"
I20260619 02:25:37.856849 10596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:25:37.859225 10611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:37.860508 10596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:25:37.860750 10596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "953452fb0a20456ba1dbb9a0112db9b9"
format_stamp: "Formatted at 2026-06-19 02:25:37 on dist-test-slave-xkrt"
server_key: "bc375a0e659346d8f10314725b5fe761"
server_key_iv: "2642b2a9abedc7efdb8fdd5fb2895323"
server_key_version: "encryptionkey@0"
I20260619 02:25:37.860863 10596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:37.872277 10596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:37.875025 10596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:37.875218 10596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:37.881937 10596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44089
I20260619 02:25:37.882205 10663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44089 every 8 connection(s)
I20260619 02:25:37.882885 10596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:25:37.885465 10664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:37.889734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 10596
I20260619 02:25:37.889865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:25:37.890147 392 external_mini_cluster.cc:1428] Setting key 961d70244fb96cf2db293e587175cd4b
I20260619 02:25:37.890862 10664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9: Bootstrap starting.
I20260619 02:25:37.893100 10664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:37.893857 10664 log.cc:824] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:37.895876 10664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9: No bootstrap required, opened a new log
Jun 19 02:25:37 dist-test-slave-xkrt krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:25:37.899994 10664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953452fb0a20456ba1dbb9a0112db9b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44089 } }
I20260619 02:25:37.900224 10664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:37.900301 10664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 953452fb0a20456ba1dbb9a0112db9b9, State: Initialized, Role: FOLLOWER
I20260619 02:25:37.900810 10664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953452fb0a20456ba1dbb9a0112db9b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44089 } }
I20260619 02:25:37.900998 10664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:25:37.901062 10664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:25:37.901153 10664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:37.902094 10664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953452fb0a20456ba1dbb9a0112db9b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44089 } }
I20260619 02:25:37.902454 10664 leader_election.cc:302] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 953452fb0a20456ba1dbb9a0112db9b9; no voters:
I20260619 02:25:37.902760 10664 leader_election.cc:288] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:25:37.902825 10669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:37.903019 10669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [term 1 LEADER]: Becoming Leader. State: Replica: 953452fb0a20456ba1dbb9a0112db9b9, State: Running, Role: LEADER
I20260619 02:25:37.903316 10669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953452fb0a20456ba1dbb9a0112db9b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44089 } }
I20260619 02:25:37.903991 10664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:25:37.904417 10669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "953452fb0a20456ba1dbb9a0112db9b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953452fb0a20456ba1dbb9a0112db9b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44089 } } }
I20260619 02:25:37.904536 10669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:37.904961 10670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 953452fb0a20456ba1dbb9a0112db9b9. Latest consensus state: current_term: 1 leader_uuid: "953452fb0a20456ba1dbb9a0112db9b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953452fb0a20456ba1dbb9a0112db9b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44089 } } }
I20260619 02:25:37.905107 10670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:25:37.905367 10673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:25:37.905426 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:37.891487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.0.98.62:44089)
0619 02:25:37.891881 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:37.891895 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:37.891929 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:25:37.892598 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:37.892615 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:37.892938 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:37.893262 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:37.894112 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:37.895486 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:37.896405 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:37.897064 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:37.897223 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:37.899909 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:37.899914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:37.899926 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:37.899950 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:37.901975 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:37.902504 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:37.902509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:37.902513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:37.902684 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:37.902998 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:37.903002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:37.903003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:37.903603 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:37.903809 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:37.904087 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 02:25:37.904395 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":119,"threads_started":1}
I20260619 02:25:37.908273 10673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:25:37.913677 10673 catalog_manager.cc:1347] Generated new cluster ID: 12a1492a0ba54edb8d63be8b56dfa25b
I20260619 02:25:37.913736 10673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:25:37.935225 10673 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:25:37.936210 10673 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:25:37.945456 10673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9: Generated new TSK 0
I20260619 02:25:37.946175 10673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:25:37.972918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44089
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:38.087035 10690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:38.087332 10690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:38.087420 10690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:38.091499 10690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:38.091580 10690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:38.091667 10690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:25:38.097009 10690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:38.098098 10690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:38.099097 10690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:38.108685 10696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:38.108686 10698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:38.108706 10695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:38.109048 10690 server_base.cc:1034] running on GCE node
I20260619 02:25:38.109454 10690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:38.110059 10690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:38.111248 10690 hybrid_clock.cc:648] HybridClock initialized: now 1781835938111217 us; error 54 us; skew 500 ppm
Jun 19 02:25:38 dist-test-slave-xkrt krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:38.113969 10690 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:25:38.114732 10690 webserver.cc:460] Webserver started at http://127.0.98.1:35229/ using document root <none> and password file <none>
I20260619 02:25:38.115226 10690 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:38.115283 10690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:38.115445 10690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:38.117292 10690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "7f74adf15fbf4391905935750fc92536"
format_stamp: "Formatted at 2026-06-19 02:25:38 on dist-test-slave-xkrt"
server_key: "7111e39ce44c6dc4d63b5051f334830e"
server_key_iv: "3436dc5ee59f580878dff836081fa7bc"
server_key_version: "encryptionkey@0"
I20260619 02:25:38.117853 10690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7f74adf15fbf4391905935750fc92536"
format_stamp: "Formatted at 2026-06-19 02:25:38 on dist-test-slave-xkrt"
server_key: "7111e39ce44c6dc4d63b5051f334830e"
server_key_iv: "3436dc5ee59f580878dff836081fa7bc"
server_key_version: "encryptionkey@0"
I20260619 02:25:38.121222 10690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 02:25:38.123533 10705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:38.124720 10690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:38.124809 10690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "7f74adf15fbf4391905935750fc92536"
format_stamp: "Formatted at 2026-06-19 02:25:38 on dist-test-slave-xkrt"
server_key: "7111e39ce44c6dc4d63b5051f334830e"
server_key_iv: "3436dc5ee59f580878dff836081fa7bc"
server_key_version: "encryptionkey@0"
I20260619 02:25:38.124888 10690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:38.146252 10690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:38.149143 10690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:38.149297 10690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:38.149852 10690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:38.150624 10690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:38.150676 10690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:38.150715 10690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:38.150730 10690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:38.159667 10690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35671
I20260619 02:25:38.159708 10818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35671 every 8 connection(s)
I20260619 02:25:38.160538 10690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:25:38.166318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 10690
I20260619 02:25:38.166445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:25:38.166709 392 external_mini_cluster.cc:1428] Setting key 5b3bc9b6ce6647eefc117a7bd91ea924
Jun 19 02:25:38 dist-test-slave-xkrt krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:25:38.174700 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.162273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60289 (local address 127.0.98.62:44089)
0619 02:25:38.162410 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:38.162419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:38.163632 (+ 1213us) server_negotiation.cc:407] Connection header received
0619 02:25:38.164535 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:38.164537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:38.164583 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:38.164661 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:38.165866 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.166405 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.167890 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.168220 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.168329 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.170843 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:38.170846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:38.170848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:38.170870 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:38.172364 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:38.173001 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:38.173004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:38.173005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:38.173088 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:38.173504 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:38.173507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:38.173508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:38.173739 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:38.173819 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:38.174460 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 02:25:38.174568 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:25:38.175837 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.162590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44089 (local address 127.0.98.1:60289)
0619 02:25:38.163477 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:38.163521 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:38.164305 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:38.164820 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:38.164826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:38.165169 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:38.165731 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.165745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.166591 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.166594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.167728 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.167738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.168361 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.168364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.168505 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.169120 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:38.169135 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:38.170694 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:38.172478 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:38.172485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:38.172501 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:38.172874 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:38.173194 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:38.173199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:38.173202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:38.173403 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:38.173846 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:38.173853 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:38.174167 (+ 314us) client_negotiation.cc:769] Sending connection context
0619 02:25:38.174380 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 02:25:38.174741 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":525,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:25:38.177058 10819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44089
I20260619 02:25:38.177333 10819 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:38.177897 10819 heartbeater.cc:511] Master 127.0.98.62:44089 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 02:25:38.179612 10628 ts_manager.cc:194] Registered new tserver with Master: 7f74adf15fbf4391905935750fc92536 (127.0.98.1:35671)
I20260619 02:25:38.180719 10628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:25:38.194777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44089
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:38.301404 10824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:38.301723 10824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:38.301803 10824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:38.306187 10824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:38.306272 10824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:38.306377 10824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:25:38.311442 10824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:38.312436 10824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:38.313401 10824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:38.322557 10832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:38.322566 10829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:38.322623 10830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:38.322630 10824 server_base.cc:1034] running on GCE node
I20260619 02:25:38.323215 10824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:38.323802 10824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:38.324966 10824 hybrid_clock.cc:648] HybridClock initialized: now 1781835938324950 us; error 27 us; skew 500 ppm
Jun 19 02:25:38 dist-test-slave-xkrt krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:38.327559 10824 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:25:38.328393 10824 webserver.cc:460] Webserver started at http://127.0.98.2:40643/ using document root <none> and password file <none>
I20260619 02:25:38.328931 10824 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:38.329001 10824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:38.329161 10824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:38.331292 10824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "3c328d34da8e4f46b4aac03ea4bba8ef"
format_stamp: "Formatted at 2026-06-19 02:25:38 on dist-test-slave-xkrt"
server_key: "ffbfe1f2428303b2cc0d1fe5ddb277da"
server_key_iv: "e2e06bcc73d1a27c519f96828abe90ff"
server_key_version: "encryptionkey@0"
I20260619 02:25:38.331796 10824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3c328d34da8e4f46b4aac03ea4bba8ef"
format_stamp: "Formatted at 2026-06-19 02:25:38 on dist-test-slave-xkrt"
server_key: "ffbfe1f2428303b2cc0d1fe5ddb277da"
server_key_iv: "e2e06bcc73d1a27c519f96828abe90ff"
server_key_version: "encryptionkey@0"
I20260619 02:25:38.335336 10824 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:25:38.337781 10839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:38.338984 10824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:38.339126 10824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "3c328d34da8e4f46b4aac03ea4bba8ef"
format_stamp: "Formatted at 2026-06-19 02:25:38 on dist-test-slave-xkrt"
server_key: "ffbfe1f2428303b2cc0d1fe5ddb277da"
server_key_iv: "e2e06bcc73d1a27c519f96828abe90ff"
server_key_version: "encryptionkey@0"
I20260619 02:25:38.339236 10824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:38.360641 10824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:38.363667 10824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:38.363864 10824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:38.364418 10824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:38.365217 10824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:38.365288 10824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:38.365375 10824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:38.365419 10824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:38.374426 10824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44505
I20260619 02:25:38.374445 10952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44505 every 8 connection(s)
I20260619 02:25:38.375295 10824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:25:38.380504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 10824
I20260619 02:25:38.380633 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:25:38.380959 392 external_mini_cluster.cc:1428] Setting key d595cbd868a92998e62735cff7985df0
Jun 19 02:25:38 dist-test-slave-xkrt krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:25:38.390045 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.377022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52921 (local address 127.0.98.62:44089)
0619 02:25:38.377188 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:38.377193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:38.378528 (+ 1335us) server_negotiation.cc:407] Connection header received
0619 02:25:38.379425 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:38.379428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:38.379481 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:38.379563 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:38.380849 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.381490 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.383010 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.383216 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.383299 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.386053 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:38.386060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:38.386066 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:38.386098 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:38.387855 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:38.388372 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:38.388378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:38.388381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:38.388478 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:38.388872 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:38.388877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:38.388879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:38.389116 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:38.389223 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:38.389720 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 02:25:38.389844 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:25:38.390733 10955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.377420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44089 (local address 127.0.98.2:52921)
0619 02:25:38.378337 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:38.378381 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:38.379227 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:38.379719 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:38.379725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:38.380062 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:38.380685 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.380705 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.381676 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.381680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.382886 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.382899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.383375 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.383382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.383611 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.384216 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:38.384231 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:38.385882 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:38.387997 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:38.388001 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:38.388011 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:38.388265 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:38.388611 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:38.388615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:38.388617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:38.388759 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:38.389220 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:38.389225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:38.389442 (+ 217us) client_negotiation.cc:769] Sending connection context
0619 02:25:38.389647 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 02:25:38.389898 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":581,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:25:38.391885 10953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44089
I20260619 02:25:38.392122 10953 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:38.392632 10953 heartbeater.cc:511] Master 127.0.98.62:44089 requested a full tablet report, sending...
I20260619 02:25:38.393641 10628 ts_manager.cc:194] Registered new tserver with Master: 3c328d34da8e4f46b4aac03ea4bba8ef (127.0.98.2:44505)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 02:25:38.394349 10628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:25:38.408581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44089
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:25:38.518285 10958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:25:38.518527 10958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:25:38.518567 10958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:25:38.522632 10958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:25:38.522687 10958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:25:38.522743 10958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:25:38.527716 10958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:25:38.528650 10958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:25:38.529624 10958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:25:38.538849 10966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:38.539000 10958 server_base.cc:1034] running on GCE node
W20260619 02:25:38.538828 10964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:25:38.539196 10963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:25:38.539593 10958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:25:38.540185 10958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:25:38.541345 10958 hybrid_clock.cc:648] HybridClock initialized: now 1781835938541309 us; error 46 us; skew 500 ppm
Jun 19 02:25:38 dist-test-slave-xkrt krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:25:38.544266 10958 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:25:38.545410 10958 webserver.cc:460] Webserver started at http://127.0.98.3:45871/ using document root <none> and password file <none>
I20260619 02:25:38.546016 10958 fs_manager.cc:359] Metadata directory not provided
I20260619 02:25:38.546077 10958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:25:38.546334 10958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:25:38.549116 10958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "8998fa052c734dd6b3ecd7dce04bc2d7"
format_stamp: "Formatted at 2026-06-19 02:25:38 on dist-test-slave-xkrt"
server_key: "7cfa9955e271ca78ebc0a7b1cafe9c03"
server_key_iv: "edec2831a993999bf7435bf79466a57f"
server_key_version: "encryptionkey@0"
I20260619 02:25:38.549743 10958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8998fa052c734dd6b3ecd7dce04bc2d7"
format_stamp: "Formatted at 2026-06-19 02:25:38 on dist-test-slave-xkrt"
server_key: "7cfa9955e271ca78ebc0a7b1cafe9c03"
server_key_iv: "edec2831a993999bf7435bf79466a57f"
server_key_version: "encryptionkey@0"
I20260619 02:25:38.553596 10958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:25:38.555936 10973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:38.557183 10958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:25:38.557319 10958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "8998fa052c734dd6b3ecd7dce04bc2d7"
format_stamp: "Formatted at 2026-06-19 02:25:38 on dist-test-slave-xkrt"
server_key: "7cfa9955e271ca78ebc0a7b1cafe9c03"
server_key_iv: "edec2831a993999bf7435bf79466a57f"
server_key_version: "encryptionkey@0"
I20260619 02:25:38.557433 10958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:25:38.572515 10958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:25:38.575413 10958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:25:38.575613 10958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:25:38.576165 10958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:25:38.576953 10958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:25:38.577026 10958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:38.577092 10958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:25:38.577137 10958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:25:38.586102 10958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33379
I20260619 02:25:38.586139 11086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33379 every 8 connection(s)
I20260619 02:25:38.586948 10958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:25:38 dist-test-slave-xkrt krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:25:38.596046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 10958
I20260619 02:25:38.596191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:25:38.596487 392 external_mini_cluster.cc:1428] Setting key 56d0b37fc85be052c1ea8d9be0d4b629
I20260619 02:25:38.600126 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.588688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55451 (local address 127.0.98.62:44089)
0619 02:25:38.588815 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:38.588819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:38.589596 (+ 777us) server_negotiation.cc:407] Connection header received
0619 02:25:38.590882 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:38.590886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:38.590943 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:38.591046 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:38.592150 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.592681 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.593829 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.594046 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.594154 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.596425 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:38.596428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:38.596430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:38.596453 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:38.598223 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:38.598854 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:38.598857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:38.598858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:38.598931 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:38.599208 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:38.599210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:38.599212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:38.599369 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:38.599443 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:38.599909 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 02:25:38.600000 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:25:38.600956 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.589002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44089 (local address 127.0.98.3:55451)
0619 02:25:38.589415 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:38.589454 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:38.590685 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:38.591223 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:38.591230 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:38.591568 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:25:38.592007 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.592029 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.592798 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.592802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.593716 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.593724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.594186 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.594189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.594351 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.594859 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:25:38.594874 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:25:38.596261 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:25:38.598353 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:38.598357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:38.598367 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:38.598767 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:38.599007 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:25:38.599009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:25:38.599011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:25:38.599135 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:25:38.599483 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:25:38.599488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:25:38.599712 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 02:25:38.599872 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 02:25:38.600117 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":104,"threads_started":1}
I20260619 02:25:38.601991 11087 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44089
I20260619 02:25:38.602209 11087 heartbeater.cc:464] Registering TS with master...
I20260619 02:25:38.602699 11087 heartbeater.cc:511] Master 127.0.98.62:44089 requested a full tablet report, sending...
I20260619 02:25:38.603648 10628 ts_manager.cc:194] Registered new tserver with Master: 8998fa052c734dd6b3ecd7dce04bc2d7 (127.0.98.3:33379)
I20260619 02:25:38.604194 10628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55451
I20260619 02:25:38.610061 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:25:38.618368 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.611977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.0.98.62:44089)
0619 02:25:38.612086 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:38.612089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:38.612230 (+ 141us) server_negotiation.cc:407] Connection header received
0619 02:25:38.612312 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:38.612314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:38.612347 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:38.612407 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:25:38.612980 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.613453 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.614257 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.614451 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.614538 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.615690 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:25:38.615692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:25:38.615694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:25:38.615716 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:25:38.616928 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:38.617305 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:38.617307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:38.617308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:38.617372 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:25:38.617641 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:25:38.617643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:25:38.617644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:25:38.617821 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:25:38.617919 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:38.618172 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 02:25:38.618261 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:25:38.620813 10628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:25:38.623716 10628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:25:38.637490 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.632562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33379 (local address 127.0.0.1:51524)
0619 02:25:38.632924 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:38.632938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:38.633027 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:38.633840 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:38.633843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:38.633858 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:38.633940 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.633944 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.634882 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.634888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.636008 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.636024 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.636912 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.636916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.637030 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.637071 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 02:25:38.637188 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 02:25:38.637348 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":101,"threads_started":1}
I20260619 02:25:38.637482 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.632692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.0.98.3:33379)
0619 02:25:38.633284 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:38.633292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:38.633309 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:25:38.633364 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:38.633367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:38.633536 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:38.633689 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:38.634039 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.634765 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.636131 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.636786 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.636890 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.637021 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:38.637227 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 02:25:38.637320 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":94,"threads_started":1}
I20260619 02:25:38.637853 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.631939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35671 (local address 127.0.0.1:42644)
0619 02:25:38.632972 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:38.632986 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:38.633082 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:38.633710 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:38.633713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:38.633734 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:38.633843 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.633849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.635326 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.635329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.636449 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.636457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.637403 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.637407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.637522 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.637574 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 02:25:38.637630 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:25:38.637697 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260619 02:25:38.638131 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.631838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44505 (local address 127.0.0.1:33852)
0619 02:25:38.632151 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:38.632194 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:38.633000 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:38.633534 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:38.633577 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:38.633602 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:38.633741 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.633748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.635203 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.635207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.636422 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.636432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.637743 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.637746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.637849 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.637870 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:25:38.637933 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 02:25:38.637999 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260619 02:25:38.638587 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.631959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.0.98.2:44505)
0619 02:25:38.632987 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:38.633000 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:38.633023 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:25:38.633165 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:38.633170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:38.633374 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:38.633521 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:38.633853 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.635061 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.636551 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.637305 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.637403 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.637527 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:38.638340 (+ 813us) server_negotiation.cc:299] Negotiation successful
0619 02:25:38.638436 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":98,"threads_started":1}
I20260619 02:25:38.638949 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.632604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.0.98.1:35671)
0619 02:25:38.633264 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:38.633278 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:38.633297 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:25:38.633361 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:38.633366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:38.633562 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:38.633985 (+ 423us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:38.634151 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.635201 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.636570 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.637256 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.638559 (+ 1303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.638685 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:38.638731 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 02:25:38.638810 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":75,"threads_started":1}
I20260619 02:25:38.640728 10887 tablet_service.cc:1467] Processing CreateTablet for tablet 3c062414693d4452a8a2f00d1038fba8 (DEFAULT_TABLE table=test-table [id=60769048d722415cabe83e2b82468221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:38.640641 11021 tablet_service.cc:1467] Processing CreateTablet for tablet 3c062414693d4452a8a2f00d1038fba8 (DEFAULT_TABLE table=test-table [id=60769048d722415cabe83e2b82468221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:38.641678 10887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c062414693d4452a8a2f00d1038fba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:38.641304 10753 tablet_service.cc:1467] Processing CreateTablet for tablet 3c062414693d4452a8a2f00d1038fba8 (DEFAULT_TABLE table=test-table [id=60769048d722415cabe83e2b82468221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:25:38.641709 11021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c062414693d4452a8a2f00d1038fba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:38.642174 10753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c062414693d4452a8a2f00d1038fba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:25:38.647738 11104 tablet_bootstrap.cc:492] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef: Bootstrap starting.
I20260619 02:25:38.648898 11105 tablet_bootstrap.cc:492] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7: Bootstrap starting.
I20260619 02:25:38.649816 11106 tablet_bootstrap.cc:492] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536: Bootstrap starting.
I20260619 02:25:38.650496 11104 tablet_bootstrap.cc:654] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:38.650978 11105 tablet_bootstrap.cc:654] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:38.651419 11104 log.cc:824] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:38.651615 11106 tablet_bootstrap.cc:654] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536: Neither blocks nor log segments found. Creating new log.
I20260619 02:25:38.652076 11105 log.cc:824] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:38.652786 11106 log.cc:824] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536: Log is configured to *not* fsync() on all Append() calls
I20260619 02:25:38.653745 11104 tablet_bootstrap.cc:492] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef: No bootstrap required, opened a new log
I20260619 02:25:38.653748 11105 tablet_bootstrap.cc:492] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7: No bootstrap required, opened a new log
I20260619 02:25:38.653928 11104 ts_tablet_manager.cc:1397] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:25:38.653928 11105 ts_tablet_manager.cc:1397] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260619 02:25:38.654505 11106 tablet_bootstrap.cc:492] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536: No bootstrap required, opened a new log
I20260619 02:25:38.654690 11106 ts_tablet_manager.cc:1397] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:25:38.657269 11106 raft_consensus.cc:348] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c328d34da8e4f46b4aac03ea4bba8ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44505 } } peers { permanent_uuid: "7f74adf15fbf4391905935750fc92536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260619 02:25:38.657506 11106 raft_consensus.cc:374] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:38.657575 11106 raft_consensus.cc:729] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f74adf15fbf4391905935750fc92536, State: Initialized, Role: FOLLOWER
I20260619 02:25:38.657702 11104 raft_consensus.cc:348] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c328d34da8e4f46b4aac03ea4bba8ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44505 } } peers { permanent_uuid: "7f74adf15fbf4391905935750fc92536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260619 02:25:38.657702 11105 raft_consensus.cc:348] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c328d34da8e4f46b4aac03ea4bba8ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44505 } } peers { permanent_uuid: "7f74adf15fbf4391905935750fc92536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260619 02:25:38.657956 11104 raft_consensus.cc:374] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:38.657956 11105 raft_consensus.cc:374] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:25:38.658012 11104 raft_consensus.cc:729] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c328d34da8e4f46b4aac03ea4bba8ef, State: Initialized, Role: FOLLOWER
I20260619 02:25:38.658012 11105 raft_consensus.cc:729] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8998fa052c734dd6b3ecd7dce04bc2d7, State: Initialized, Role: FOLLOWER
I20260619 02:25:38.658032 11106 consensus_queue.cc:260] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c328d34da8e4f46b4aac03ea4bba8ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44505 } } peers { permanent_uuid: "7f74adf15fbf4391905935750fc92536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260619 02:25:38.658455 11105 consensus_queue.cc:260] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c328d34da8e4f46b4aac03ea4bba8ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44505 } } peers { permanent_uuid: "7f74adf15fbf4391905935750fc92536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260619 02:25:38.658455 11104 consensus_queue.cc:260] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c328d34da8e4f46b4aac03ea4bba8ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44505 } } peers { permanent_uuid: "7f74adf15fbf4391905935750fc92536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260619 02:25:38.658634 10819 heartbeater.cc:503] Master 127.0.98.62:44089 was elected leader, sending a full tablet report...
I20260619 02:25:38.658989 11106 ts_tablet_manager.cc:1428] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 02:25:38.659343 11105 ts_tablet_manager.cc:1428] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260619 02:25:38.659644 11087 heartbeater.cc:503] Master 127.0.98.62:44089 was elected leader, sending a full tablet report...
I20260619 02:25:38.660054 11104 ts_tablet_manager.cc:1428] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:25:38.660349 10953 heartbeater.cc:503] Master 127.0.98.62:44089 was elected leader, sending a full tablet report...
W20260619 02:25:38.661594 10820 tablet.cc:2369] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:25:38.804301 11111 raft_consensus.cc:482] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:25:38.804497 11111 raft_consensus.cc:504] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c328d34da8e4f46b4aac03ea4bba8ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44505 } } peers { permanent_uuid: "7f74adf15fbf4391905935750fc92536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260619 02:25:38.805804 11111 leader_election.cc:288] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c328d34da8e4f46b4aac03ea4bba8ef (127.0.98.2:44505), 7f74adf15fbf4391905935750fc92536 (127.0.98.1:35671)
I20260619 02:25:38.809521 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.806133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45935 (local address 127.0.98.2:44505)
0619 02:25:38.806245 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:38.806252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:38.806299 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:25:38.806496 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:38.806501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:38.806570 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:38.806659 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:38.806954 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.807519 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.808544 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.808959 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.809075 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.809137 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:38.809312 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 02:25:38.809388 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:25:38.809530 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.806034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44505 (local address 127.0.98.3:45935)
0619 02:25:38.806220 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:38.806235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:38.806348 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:38.806703 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:38.806707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:38.806725 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:38.806841 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.806846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.807632 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.807638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.808422 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.808430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.809076 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.809079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.809190 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.809209 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:25:38.809298 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 02:25:38.809407 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":21}
I20260619 02:25:38.809569 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.806176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41899 (local address 127.0.98.1:35671)
0619 02:25:38.806322 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:38.806327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:38.806548 (+ 221us) server_negotiation.cc:407] Connection header received
0619 02:25:38.806671 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:38.806674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:38.806726 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:38.806806 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:25:38.807052 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.807593 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.808559 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.808946 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.809057 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.809119 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:38.809276 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 02:25:38.809348 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:25:38.809551 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.806097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35671 (local address 127.0.98.3:41899)
0619 02:25:38.806461 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:38.806472 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:25:38.806576 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:25:38.806815 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:25:38.806818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:25:38.806833 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:25:38.806949 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.806954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.807696 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.807701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.808442 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:25:38.808450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.809062 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.809066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:25:38.809158 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.809184 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 02:25:38.809264 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 02:25:38.809346 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":3,"thread_start_us":151,"threads_started":1}
I20260619 02:25:38.810245 10773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c062414693d4452a8a2f00d1038fba8" candidate_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f74adf15fbf4391905935750fc92536" is_pre_election: true
I20260619 02:25:38.810246 10907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c062414693d4452a8a2f00d1038fba8" candidate_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c328d34da8e4f46b4aac03ea4bba8ef" is_pre_election: true
I20260619 02:25:38.810549 10907 raft_consensus.cc:2454] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8998fa052c734dd6b3ecd7dce04bc2d7 in term 0.
I20260619 02:25:38.810549 10773 raft_consensus.cc:2454] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8998fa052c734dd6b3ecd7dce04bc2d7 in term 0.
I20260619 02:25:38.811010 10974 leader_election.cc:302] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c328d34da8e4f46b4aac03ea4bba8ef, 8998fa052c734dd6b3ecd7dce04bc2d7; no voters:
I20260619 02:25:38.811277 11111 raft_consensus.cc:2781] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:25:38.811368 11111 raft_consensus.cc:482] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:25:38.811424 11111 raft_consensus.cc:3037] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:38.812474 11111 raft_consensus.cc:504] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c328d34da8e4f46b4aac03ea4bba8ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44505 } } peers { permanent_uuid: "7f74adf15fbf4391905935750fc92536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260619 02:25:38.812813 11111 leader_election.cc:288] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [CANDIDATE]: Term 1 election: Requested vote from peers 3c328d34da8e4f46b4aac03ea4bba8ef (127.0.98.2:44505), 7f74adf15fbf4391905935750fc92536 (127.0.98.1:35671)
I20260619 02:25:38.813129 10907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c062414693d4452a8a2f00d1038fba8" candidate_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c328d34da8e4f46b4aac03ea4bba8ef"
I20260619 02:25:38.813148 10773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c062414693d4452a8a2f00d1038fba8" candidate_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f74adf15fbf4391905935750fc92536"
I20260619 02:25:38.813257 10907 raft_consensus.cc:3037] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:38.813279 10773 raft_consensus.cc:3037] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:25:38.814322 10773 raft_consensus.cc:2454] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8998fa052c734dd6b3ecd7dce04bc2d7 in term 1.
I20260619 02:25:38.814402 10907 raft_consensus.cc:2454] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8998fa052c734dd6b3ecd7dce04bc2d7 in term 1.
I20260619 02:25:38.814603 10976 leader_election.cc:302] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f74adf15fbf4391905935750fc92536, 8998fa052c734dd6b3ecd7dce04bc2d7; no voters:
I20260619 02:25:38.814780 11111 raft_consensus.cc:2781] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:25:38.815050 11111 raft_consensus.cc:686] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [term 1 LEADER]: Becoming Leader. State: Replica: 8998fa052c734dd6b3ecd7dce04bc2d7, State: Running, Role: LEADER
I20260619 02:25:38.815390 11111 consensus_queue.cc:237] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c328d34da8e4f46b4aac03ea4bba8ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44505 } } peers { permanent_uuid: "7f74adf15fbf4391905935750fc92536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260619 02:25:38.818221 10626 catalog_manager.cc:5495] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8998fa052c734dd6b3ecd7dce04bc2d7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c328d34da8e4f46b4aac03ea4bba8ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f74adf15fbf4391905935750fc92536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8998fa052c734dd6b3ecd7dce04bc2d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } health_report { overall_health: HEALTHY } } }
W20260619 02:25:38.837823 11088 tablet.cc:2369] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:25:38.869277 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.865762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51538 (local address 127.0.98.3:33379)
0619 02:25:38.865897 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:25:38.865904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:25:38.865920 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:25:38.866109 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:25:38.866112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:25:38.866167 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:25:38.866272 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:25:38.866513 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.867006 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.867873 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:25:38.868097 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:25:38.868200 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:25:38.868489 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:25:38.868901 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:25:38.869036 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:25:38.869072 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 02:25:38.869122 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:25:38.874785 10773 raft_consensus.cc:1261] T 3c062414693d4452a8a2f00d1038fba8 P 7f74adf15fbf4391905935750fc92536 [term 1 FOLLOWER]: Refusing update from remote peer 8998fa052c734dd6b3ecd7dce04bc2d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:38.874785 10907 raft_consensus.cc:1261] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef [term 1 FOLLOWER]: Refusing update from remote peer 8998fa052c734dd6b3ecd7dce04bc2d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:25:38.875479 11114 consensus_queue.cc:1035] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f74adf15fbf4391905935750fc92536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:25:38.875658 11111 consensus_queue.cc:1035] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c328d34da8e4f46b4aac03ea4bba8ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:25:38.876293 10954 tablet.cc:2369] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:25:38.883008 11117 mvcc.cc:205] Tried to move back new op lower bound from 7298400005624045568 to 7298400005390434304. Current Snapshot: MvccSnapshot[applied={T|T < 7298400005624045568}]
I20260619 02:25:38.890477 10627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:38.890646 10627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:25:38.892871 10627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 953452fb0a20456ba1dbb9a0112db9b9: Sending DeleteTablet for 3 replicas of tablet 3c062414693d4452a8a2f00d1038fba8
I20260619 02:25:38.893501 10887 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c062414693d4452a8a2f00d1038fba8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33852
I20260619 02:25:38.893724 10753 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c062414693d4452a8a2f00d1038fba8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42644
I20260619 02:25:38.893870 11021 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c062414693d4452a8a2f00d1038fba8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:25:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51524
I20260619 02:25:38.894253 11129 tablet_replica.cc:330] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7: stopping tablet replica
I20260619 02:25:38.894536 11129 raft_consensus.cc:2229] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:25:38.894845 11129 raft_consensus.cc:2258] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:38.895056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 10690
I20260619 02:25:38.896411 11129 ts_tablet_manager.cc:1905] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:38.899623 11129 ts_tablet_manager.cc:1918] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:38.899730 11129 log.cc:1196] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/3c062414693d4452a8a2f00d1038fba8
I20260619 02:25:38.899966 11127 tablet_replica.cc:330] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef: stopping tablet replica
I20260619 02:25:38.900040 11129 ts_tablet_manager.cc:1939] T 3c062414693d4452a8a2f00d1038fba8 P 8998fa052c734dd6b3ecd7dce04bc2d7: Deleting consensus metadata
I20260619 02:25:38.900333 11127 raft_consensus.cc:2229] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:25:38.900679 11127 raft_consensus.cc:2258] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:25:38.901072 10615 catalog_manager.cc:4841] TS 8998fa052c734dd6b3ecd7dce04bc2d7 (127.0.98.3:33379): tablet 3c062414693d4452a8a2f00d1038fba8 (table test-table [id=60769048d722415cabe83e2b82468221]) successfully deleted
I20260619 02:25:38.902051 11127 ts_tablet_manager.cc:1905] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:25:38.904430 11127 ts_tablet_manager.cc:1918] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:25:38.904526 11127 log.cc:1196] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/3c062414693d4452a8a2f00d1038fba8
I20260619 02:25:38.904840 11127 ts_tablet_manager.cc:1939] T 3c062414693d4452a8a2f00d1038fba8 P 3c328d34da8e4f46b4aac03ea4bba8ef: Deleting consensus metadata
I20260619 02:25:38.905908 10612 catalog_manager.cc:4841] TS 3c328d34da8e4f46b4aac03ea4bba8ef (127.0.98.2:44505): tablet 3c062414693d4452a8a2f00d1038fba8 (table test-table [id=60769048d722415cabe83e2b82468221]) successfully deleted
W20260619 02:25:38.906742 10614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35671 (error 108)
I20260619 02:25:38.907034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 10824
I20260619 02:25:38.910202 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:25:38.909933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35671 (local address 127.0.0.1:42648)
0619 02:25:38.910066 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 02:25:38.910139 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 02:25:38.910328 10614 catalog_manager.cc:4584] TS 7f74adf15fbf4391905935750fc92536 (127.0.98.1:35671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c062414693d4452a8a2f00d1038fba8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35671: connect: Connection refused (error 111)
I20260619 02:25:38.914217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 10958
I20260619 02:25:38.919652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 10596
2026-06-19T02:25:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2353 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:25:38 dist-test-slave-xkrt krb5kdc[11133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:25:38 dist-test-slave-xkrt krb5kdc[11133](info): set up 2 sockets
Jun 19 02:25:38 dist-test-slave-xkrt krb5kdc[11133](info): commencing operation
krb5kdc: starting...
W20260619 02:25:39.954664 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:25:40 dist-test-slave-xkrt krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:25:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:25:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 02:25:40.121053 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjWwzrN/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/postgres -l logfile start
2026-06-19 02:25:42.533 UTC [11163] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 02:25:42.533 UTC [11163] LOG: listening on IPv4 address "127.0.98.20", port 34963
2026-06-19 02:25:42.537 UTC [11163] LOG: listening on Unix socket "/tmp/.s.PGSQL.34963"
2026-06-19 02:25:42.561 UTC [11166] LOG: database system was shut down at 2026-06-19 02:25:41 UTC
2026-06-19 02:25:42.565 UTC [11163] LOG: database system is ready to accept connections
I20260619 02:25:43.518421 392 mini_postgres.cc:96] Postgres bound to 34963
2026-06-19 02:25:43.525 UTC [11174] FATAL: database "slave" does not exist
127.0.98.20:34963 - accepting connections
I20260619 02:25:43.525708 392 mini_ranger.cc:161] Starting Ranger...
I20260619 02:25:43.541016 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 02:25:43.986359 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 02:25:43.986450 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-admin
2026-06-19 02:25:44,263 [I] DB FLAVOR :POSTGRES
2026-06-19 02:25:44,263 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:25:44,263 [I] Checking connection..
2026-06-19 02:25:44,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 02:25:44,504 [I] Checking connection passed.
2026-06-19 02:25:44,504 [I] --------- Verifying version history table ---------
2026-06-19 02:25:44,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:25:44,741 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:25:44,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:25:44,970 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:25:44,970 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 02:25:44,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 02:25:45.206 UTC [11238] WARNING: there is no transaction in progress
2026-06-19 02:25:45,212 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 02:25:45,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:25:45,468 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 02:25:45,468 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:25:45,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 02:25:45,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 02:25:45,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xkrt.c.gcp-upstream.internal','N') ;"
2026-06-19 02:25:46,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:25:46,411 [I] Table x_portal_user does not exist in database ranger
2026-06-19 02:25:46,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:25:46,662 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 02:25:46,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:25:46,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:25:47,144 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 02:25:47,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 02:25:48.063 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.078 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.094 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.109 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.124 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.141 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.194 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.200 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.214 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.223 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.233 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.246 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.256 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.262 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.268 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48.274 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 02:25:48,862 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 02:25:48,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 02:25:49,099 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 02:25:49,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:25:49,322 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 02:25:49,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:25:49,546 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 02:25:49,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:25:49,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:25:50,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xkrt.c.gcp-upstream.internal';"
2026-06-19 02:25:50,262 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 02:25:50,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 02:25:50,504 [I] DB_PATCHES have already been applied
I20260619 02:25:50.509680 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjWwzrN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 02:25:50.509737 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 02:25:50.512912 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39269
Jun 19, 2026 2:25:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:25:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:25:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:25:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:25:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:25:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:25:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 02:25:51.693734 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.168s user 0.000s sys 0.009s
Jun 19, 2026 2:26:10 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:26:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 2:26:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:26:10 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:26:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:26:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:26:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:26:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:26:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:26:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:26:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:26:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:26:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:26:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:26:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:26:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:26:10 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:26:11 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 02:26:11.638810 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 02:26:11 dist-test-slave-xkrt krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835971, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 02:26:11.721740 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 02:26:11.730857 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 02:26:12.569060 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 02:26:12.569156 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-kms
I20260619 02:26:12.569257 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:34963
2026-06-19 02:26:12,756 [I] DB FLAVOR :POSTGRES
2026-06-19 02:26:12,756 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:26:12,756 [I] Checking connection
2026-06-19 02:26:12,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 02:26:13,013 [I] connection success
2026-06-19 02:26:13,013 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 02:26:13,014 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 02:26:13,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 02:26:13,251 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 02:26:13,251 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:26:13,251 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 02:26:13,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 02:26:13,498 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 02:26:13.939337 392 mini_ranger_kms.cc:310] Created kms service
I20260619 02:26:14.118410 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 02:26:14.160218 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 02:26:14 dist-test-slave-xkrt krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835971, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:26:14.529575 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 02:26:14.529740 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 02:26:14.529775 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 02:26:14.533083 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11526
I20260619 02:26:14.533231 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:44559
Jun 19, 2026 2:26:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:26:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:26:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:26:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:26:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:26:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 02:26:16.783951 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.062s user 0.001s sys 0.170s
I20260619 02:26:16.784091 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 02:26:16.784200 392 mini_ranger_kms.cc:401] 127.0.98.20:44559/kms/v1/keys
Jun 19 02:26:16 dist-test-slave-xkrt krb5kdc[11133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781835976, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 02:26:17 dist-test-slave-xkrt krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835976, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 02:26:17 dist-test-slave-xkrt krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835976, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 02:26:19 dist-test-slave-xkrt krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:26:20.033805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42831
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42831
--ranger_config_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44559/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:26:20.151785 11577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:26:20.152051 11577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:26:20.152127 11577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:26:20.156252 11577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:26:20.156315 11577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:26:20.156335 11577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:26:20.156352 11577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:26:20.156368 11577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:26:20.161978 11577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:44559/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42831
--ranger_config_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:26:20.163071 11577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:26:20.164119 11577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:26:20.174239 11582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:26:20.174229 11583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:26:20.174392 11577 server_base.cc:1034] running on GCE node
W20260619 02:26:20.174438 11585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:26:20.174916 11577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:26:20.176040 11577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:26:20.177246 11577 hybrid_clock.cc:648] HybridClock initialized: now 1781835980177230 us; error 35 us; skew 500 ppm
Jun 19 02:26:20 dist-test-slave-xkrt krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:26:20.180356 11577 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:26:20.181399 11577 webserver.cc:460] Webserver started at http://127.0.98.62:42691/ using document root <none> and password file <none>
I20260619 02:26:20.181999 11577 fs_manager.cc:359] Metadata directory not provided
I20260619 02:26:20.182086 11577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:26:20.182299 11577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:26:20 dist-test-slave-xkrt krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:26:20.242547 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "bb03991c6ad648da910bfc1432c62d7c"
format_stamp: "Formatted at 2026-06-19 02:26:20 on dist-test-slave-xkrt"
server_key: "5c51f58255f71c5138ab29c0d74a2a7f"
server_key_iv: "acfc82baacc8be0debd680b2f99e3346"
server_key_version: "kuduclusterkey@0"
I20260619 02:26:20.243197 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "bb03991c6ad648da910bfc1432c62d7c"
format_stamp: "Formatted at 2026-06-19 02:26:20 on dist-test-slave-xkrt"
server_key: "5c51f58255f71c5138ab29c0d74a2a7f"
server_key_iv: "acfc82baacc8be0debd680b2f99e3346"
server_key_version: "kuduclusterkey@0"
I20260619 02:26:20.247192 11577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:26:20.262660 11592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:26:20.264178 11577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:26:20.264340 11577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "bb03991c6ad648da910bfc1432c62d7c"
format_stamp: "Formatted at 2026-06-19 02:26:20 on dist-test-slave-xkrt"
server_key: "5c51f58255f71c5138ab29c0d74a2a7f"
server_key_iv: "acfc82baacc8be0debd680b2f99e3346"
server_key_version: "kuduclusterkey@0"
I20260619 02:26:20.264478 11577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:26:20.283438 11577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:26:20.286499 11577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:26:20.286708 11577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:26:20.294994 11577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42831
I20260619 02:26:20.295027 11644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42831 every 8 connection(s)
I20260619 02:26:20.295914 11577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:26:20.298651 11645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:26:20.301275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 11577
I20260619 02:26:20.301414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:26:20.301774 392 external_mini_cluster.cc:1428] Setting key 767bdfa87fdd367b128103eafd600055
I20260619 02:26:20.304628 11645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c: Bootstrap starting.
I20260619 02:26:20.307188 11645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c: Neither blocks nor log segments found. Creating new log.
I20260619 02:26:20.307993 11645 log.cc:824] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:26:20.310077 11645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c: No bootstrap required, opened a new log
Jun 19 02:26:20 dist-test-slave-xkrt krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:26:20.314414 11645 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb03991c6ad648da910bfc1432c62d7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42831 } }
I20260619 02:26:20.314689 11645 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:26:20.314759 11645 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb03991c6ad648da910bfc1432c62d7c, State: Initialized, Role: FOLLOWER
I20260619 02:26:20.315346 11645 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb03991c6ad648da910bfc1432c62d7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42831 } }
I20260619 02:26:20.315514 11645 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:26:20.315615 11645 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:26:20.315754 11645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:26:20.316773 11645 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb03991c6ad648da910bfc1432c62d7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42831 } }
I20260619 02:26:20.317171 11645 leader_election.cc:302] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb03991c6ad648da910bfc1432c62d7c; no voters:
I20260619 02:26:20.317571 11645 leader_election.cc:288] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:26:20.317634 11650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:26:20.317884 11650 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [term 1 LEADER]: Becoming Leader. State: Replica: bb03991c6ad648da910bfc1432c62d7c, State: Running, Role: LEADER
I20260619 02:26:20.318113 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:20.303371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.0.98.62:42831)
0619 02:26:20.303826 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 02:26:20.303844 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 02:26:20.303882 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:26:20.304672 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:26:20.304693 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:26:20.305088 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:26:20.305422 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:26:20.306278 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:20.307697 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:20.308867 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:20.309497 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:20.309696 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:20.312620 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:26:20.312626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:26:20.312644 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:26:20.312675 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:26:20.314679 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:26:20.315205 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:26:20.315211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:26:20.315217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:26:20.315357 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:26:20.315681 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:26:20.315684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:26:20.315685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:26:20.316256 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:26:20.316486 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 02:26:20.316764 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 02:26:20.317110 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":143,"threads_started":1}
I20260619 02:26:20.318437 11650 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb03991c6ad648da910bfc1432c62d7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42831 } }
I20260619 02:26:20.318866 11645 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:26:20.319684 11650 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb03991c6ad648da910bfc1432c62d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb03991c6ad648da910bfc1432c62d7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42831 } } }
I20260619 02:26:20.319886 11650 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [sys.catalog]: This master's current role is: LEADER
I20260619 02:26:20.320132 11650 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb03991c6ad648da910bfc1432c62d7c. Latest consensus state: current_term: 1 leader_uuid: "bb03991c6ad648da910bfc1432c62d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb03991c6ad648da910bfc1432c62d7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42831 } } }
I20260619 02:26:20.320266 11650 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c [sys.catalog]: This master's current role is: LEADER
I20260619 02:26:20.320593 11659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:26:20.323724 11659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:26:20.324203 11645 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:26:20.334726 11659 catalog_manager.cc:1347] Generated new cluster ID: 3b194b8328f34c6a8a2a5633dff7a77c
I20260619 02:26:20.334802 11659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:26:20.346607 11659 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:26:20.347853 11659 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:26:20.355985 11659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c: Generated new TSK 0
I20260619 02:26:20.356920 11659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:26:21.069311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42831
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:26:21.214816 11690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:26:21.215101 11690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:26:21.215200 11690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:26:21.220381 11690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:26:21.220515 11690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:26:21.220646 11690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:26:21.228650 11690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:44559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:26:21.230042 11690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:26:21.231383 11690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:26:21.243973 11698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:26:21.244146 11697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:26:21.244017 11700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:26:21.244323 11690 server_base.cc:1034] running on GCE node
I20260619 02:26:21.244735 11690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:26:21.245357 11690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:26:21.246556 11690 hybrid_clock.cc:648] HybridClock initialized: now 1781835981246536 us; error 39 us; skew 500 ppm
Jun 19 02:26:21 dist-test-slave-xkrt krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:26:21.249791 11690 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:26:21.250785 11690 webserver.cc:460] Webserver started at http://127.0.98.1:37293/ using document root <none> and password file <none>
I20260619 02:26:21.251430 11690 fs_manager.cc:359] Metadata directory not provided
I20260619 02:26:21.251515 11690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:26:21.251739 11690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:26:21 dist-test-slave-xkrt krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:26:21.273022 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "efa09dea8cf8416a8e7e50db390a99b3"
format_stamp: "Formatted at 2026-06-19 02:26:21 on dist-test-slave-xkrt"
server_key: "a0185382659ec144591ff0e5e048a758"
server_key_iv: "b9115571cb0f84ccd2c9656a32fa3435"
server_key_version: "kuduclusterkey@0"
I20260619 02:26:21.273869 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "efa09dea8cf8416a8e7e50db390a99b3"
format_stamp: "Formatted at 2026-06-19 02:26:21 on dist-test-slave-xkrt"
server_key: "a0185382659ec144591ff0e5e048a758"
server_key_iv: "b9115571cb0f84ccd2c9656a32fa3435"
server_key_version: "kuduclusterkey@0"
I20260619 02:26:21.282310 11690 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260619 02:26:21.309890 11707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:26:21.311568 11690 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.000s
I20260619 02:26:21.311751 11690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "efa09dea8cf8416a8e7e50db390a99b3"
format_stamp: "Formatted at 2026-06-19 02:26:21 on dist-test-slave-xkrt"
server_key: "a0185382659ec144591ff0e5e048a758"
server_key_iv: "b9115571cb0f84ccd2c9656a32fa3435"
server_key_version: "kuduclusterkey@0"
I20260619 02:26:21.311960 11690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:26:21.338749 11690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:26:21.342639 11690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:26:21.342824 11690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:26:21.343587 11690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:26:21.344678 11690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:26:21.344733 11690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:26:21.344787 11690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:26:21.344810 11690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:26:21.359273 11690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41303
I20260619 02:26:21.360423 11690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:26:21.362257 11822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41303 every 8 connection(s)
I20260619 02:26:21.369870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 11690
I20260619 02:26:21.370002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:26:21.370298 392 external_mini_cluster.cc:1428] Setting key 8a3279a84fb4eb6e7335dacfca628d72
Jun 19 02:26:21 dist-test-slave-xkrt krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:26:21.380923 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:21.365591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57663 (local address 127.0.98.62:42831)
0619 02:26:21.367963 (+ 2372us) server_negotiation.cc:206] Beginning negotiation
0619 02:26:21.367968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:26:21.367987 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:26:21.368608 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:26:21.368611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:26:21.368670 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:26:21.368755 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:26:21.370500 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:21.371390 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:21.373117 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:21.373533 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:21.373644 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:21.376638 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:26:21.376642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:26:21.376645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:26:21.376671 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:26:21.378495 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:26:21.379101 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:26:21.379105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:26:21.379107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:26:21.379198 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:26:21.379620 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:26:21.379623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:26:21.379625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:26:21.379836 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:26:21.379910 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:26:21.380613 (+ 703us) server_negotiation.cc:299] Negotiation successful
0619 02:26:21.380746 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2291,"thread_start_us":78,"threads_started":1}
I20260619 02:26:21.382488 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:21.364472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42831 (local address 127.0.98.1:57663)
0619 02:26:21.366342 (+ 1870us) negotiation.cc:107] Waiting for socket to connect
0619 02:26:21.366398 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:26:21.368267 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:26:21.369228 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:26:21.369239 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:26:21.369618 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:26:21.370300 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:21.370320 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:21.371556 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:21.371560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:21.372966 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:21.372977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:21.373847 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:21.373850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:21.374037 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:21.374768 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:26:21.374790 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:26:21.376448 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:26:21.378629 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:26:21.378635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:26:21.378650 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:26:21.378973 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:26:21.379331 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:26:21.379334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:26:21.379336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:26:21.379506 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:26:21.379991 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:26:21.379997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:26:21.380339 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 02:26:21.381149 (+ 810us) client_negotiation.cc:240] Negotiation successful
0619 02:26:21.381469 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1645,"thread_start_us":140,"threads_started":1}
I20260619 02:26:21.383863 11823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42831
I20260619 02:26:21.384111 11823 heartbeater.cc:464] Registering TS with master...
I20260619 02:26:21.384662 11823 heartbeater.cc:511] Master 127.0.98.62:42831 requested a full tablet report, sending...
I20260619 02:26:21.386268 11607 ts_manager.cc:194] Registered new tserver with Master: efa09dea8cf8416a8e7e50db390a99b3 (127.0.98.1:41303)
I20260619 02:26:21.387758 11607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57663
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:26:21.414397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42831
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:26:21 dist-test-slave-xkrt krb5kdc[11133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781835981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 02:26:21.568370 11829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:26:21.568804 11829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:26:21.568876 11829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:26:21.575157 11829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:26:21.575230 11829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:26:21.575316 11829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:26:21.582973 11829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:44559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:26:21.584370 11829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:26:21.585700 11829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:26:21.597918 11835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:26:21.597925 11836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:26:21.598753 11838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:26:21.599046 11829 server_base.cc:1034] running on GCE node
I20260619 02:26:21.599483 11829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:26:21.600201 11829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:26:21.601400 11829 hybrid_clock.cc:648] HybridClock initialized: now 1781835981601370 us; error 49 us; skew 500 ppm
Jun 19 02:26:21 dist-test-slave-xkrt krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:26:21.604867 11829 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:26:21.605926 11829 webserver.cc:460] Webserver started at http://127.0.98.2:35527/ using document root <none> and password file <none>
I20260619 02:26:21.606611 11829 fs_manager.cc:359] Metadata directory not provided
I20260619 02:26:21.606709 11829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:26:21.606961 11829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:26:21 dist-test-slave-xkrt krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:26:21.622354 11829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "e685b752cb2746df92754ae7368b7406"
format_stamp: "Formatted at 2026-06-19 02:26:21 on dist-test-slave-xkrt"
server_key: "61c12e3ed51144ecd02912abaed441b8"
server_key_iv: "1c3298720b1de9190048f0147c3834b2"
server_key_version: "kuduclusterkey@0"
I20260619 02:26:21.623073 11829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e685b752cb2746df92754ae7368b7406"
format_stamp: "Formatted at 2026-06-19 02:26:21 on dist-test-slave-xkrt"
server_key: "61c12e3ed51144ecd02912abaed441b8"
server_key_iv: "1c3298720b1de9190048f0147c3834b2"
server_key_version: "kuduclusterkey@0"
I20260619 02:26:21.635509 11829 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.006s sys 0.000s
I20260619 02:26:21.656471 11845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:26:21.658385 11829 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260619 02:26:21.658627 11829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "e685b752cb2746df92754ae7368b7406"
format_stamp: "Formatted at 2026-06-19 02:26:21 on dist-test-slave-xkrt"
server_key: "61c12e3ed51144ecd02912abaed441b8"
server_key_iv: "1c3298720b1de9190048f0147c3834b2"
server_key_version: "kuduclusterkey@0"
I20260619 02:26:21.658835 11829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:26:21.680836 11829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:26:21.683852 11829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:26:21.684063 11829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:26:21.684649 11829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:26:21.685485 11829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:26:21.685588 11829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:26:21.685657 11829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:26:21.685694 11829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:26:21.694763 11829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37947
I20260619 02:26:21.695691 11958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37947 every 8 connection(s)
I20260619 02:26:21.695923 11829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:26:21.698429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 11829
I20260619 02:26:21.698554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:26:21.698884 392 external_mini_cluster.cc:1428] Setting key 4beb0414ff3b6ec6fa03388184fe6b92
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 19 02:26:21 dist-test-slave-xkrt krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:26:21.736559 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:21.698010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41239 (local address 127.0.98.62:42831)
0619 02:26:21.698172 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:26:21.698177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:26:21.719705 (+ 21528us) server_negotiation.cc:407] Connection header received
0619 02:26:21.721187 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:26:21.721192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:26:21.721253 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:26:21.721336 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:26:21.724574 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:21.725422 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:21.729056 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:21.729396 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:21.729515 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:21.732363 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:26:21.732367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:26:21.732370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:26:21.732401 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:26:21.734197 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:26:21.734810 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:26:21.734814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:26:21.734816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:26:21.734905 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:26:21.735267 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:26:21.735271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:26:21.735273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:26:21.735515 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:26:21.735614 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:26:21.736248 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 02:26:21.736388 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:26:21.737782 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:21.698526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42831 (local address 127.0.98.2:41239)
0619 02:26:21.719490 (+ 20964us) negotiation.cc:107] Waiting for socket to connect
0619 02:26:21.719572 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0619 02:26:21.720907 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:26:21.723297 (+ 2390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:26:21.723307 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:26:21.723705 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:26:21.724415 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:21.724434 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:21.725643 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:21.725647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:21.728922 (+ 3275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:21.728934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:21.729606 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:21.729612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:21.729820 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:21.730494 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:26:21.730516 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:26:21.732181 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:26:21.734340 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:26:21.734347 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:26:21.734368 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:26:21.734704 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:26:21.734991 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:26:21.734994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:26:21.734996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:26:21.735167 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:26:21.735637 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:26:21.735644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:26:21.735972 (+ 328us) client_negotiation.cc:769] Sending connection context
0619 02:26:21.736183 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 02:26:21.736516 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20655,"thread_start_us":135,"threads_started":1}
I20260619 02:26:21.739077 11959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42831
I20260619 02:26:21.739336 11959 heartbeater.cc:464] Registering TS with master...
I20260619 02:26:21.739920 11959 heartbeater.cc:511] Master 127.0.98.62:42831 requested a full tablet report, sending...
I20260619 02:26:21.741044 11607 ts_manager.cc:194] Registered new tserver with Master: e685b752cb2746df92754ae7368b7406 (127.0.98.2:37947)
I20260619 02:26:21.741814 11607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:26:21.762326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42831
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:26:21 dist-test-slave-xkrt krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 02:26:21.944393 11964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:26:21.944775 11964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:26:21.944867 11964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:26:21.951428 11964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:26:21.951510 11964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:26:21.951608 11964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:26:21.959542 11964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:44559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:26:21.960875 11964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:26:21.962211 11964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:26:21.975790 11970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:26:21.976190 11972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:26:21.978557 11969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:26:21.982764 11964 server_base.cc:1034] running on GCE node
I20260619 02:26:21.983345 11964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:26:21.984180 11964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:26:21.985373 11964 hybrid_clock.cc:648] HybridClock initialized: now 1781835981985306 us; error 79 us; skew 500 ppm
Jun 19 02:26:21 dist-test-slave-xkrt krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:26:21.988910 11964 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:26:21.990203 11964 webserver.cc:460] Webserver started at http://127.0.98.3:34845/ using document root <none> and password file <none>
I20260619 02:26:21.990772 11964 fs_manager.cc:359] Metadata directory not provided
I20260619 02:26:21.990862 11964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:26:21.991071 11964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:26:21 dist-test-slave-xkrt krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:26:22.017845 11964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "bdc637f980b44affb01602174042d0d3"
format_stamp: "Formatted at 2026-06-19 02:26:22 on dist-test-slave-xkrt"
server_key: "76e6120a65cb9cdd1c318c9fe7324068"
server_key_iv: "368c82d30e53b8444c6cf829dfbefb76"
server_key_version: "kuduclusterkey@0"
I20260619 02:26:22.018813 11964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bdc637f980b44affb01602174042d0d3"
format_stamp: "Formatted at 2026-06-19 02:26:22 on dist-test-slave-xkrt"
server_key: "76e6120a65cb9cdd1c318c9fe7324068"
server_key_iv: "368c82d30e53b8444c6cf829dfbefb76"
server_key_version: "kuduclusterkey@0"
I20260619 02:26:22.027091 11964 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260619 02:26:22.062003 11979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:26:22.064011 11964 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260619 02:26:22.064142 11964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "bdc637f980b44affb01602174042d0d3"
format_stamp: "Formatted at 2026-06-19 02:26:22 on dist-test-slave-xkrt"
server_key: "76e6120a65cb9cdd1c318c9fe7324068"
server_key_iv: "368c82d30e53b8444c6cf829dfbefb76"
server_key_version: "kuduclusterkey@0"
I20260619 02:26:22.064288 11964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:26:22.082533 11964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:26:22.086256 11964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:26:22.086439 11964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:26:22.087093 11964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:26:22.088034 11964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:26:22.088095 11964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:26:22.088150 11964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:26:22.088176 11964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:26:22.101583 11964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34905
I20260619 02:26:22.102737 11964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:26:22.103478 12092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34905 every 8 connection(s)
I20260619 02:26:22.104436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 11964
I20260619 02:26:22.104549 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:26:22.104840 392 external_mini_cluster.cc:1428] Setting key 5ccc38204fe1b6f7361ba6b5cd186a42
Jun 19 02:26:22 dist-test-slave-xkrt krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:26:22.125867 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.109409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45079 (local address 127.0.98.62:42831)
0619 02:26:22.109636 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 02:26:22.109640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:26:22.111020 (+ 1380us) server_negotiation.cc:407] Connection header received
0619 02:26:22.112201 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:26:22.112205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:26:22.112258 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:26:22.112308 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:26:22.113936 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.114792 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.116478 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.116812 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.116884 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:22.121606 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:26:22.121610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:26:22.121613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:26:22.121641 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:26:22.123402 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:26:22.124034 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:26:22.124037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:26:22.124039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:26:22.124119 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:26:22.124546 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:26:22.124549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:26:22.124551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:26:22.124772 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:26:22.124848 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:26:22.125529 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 02:26:22.125698 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:26:22.127764 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.109224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42831 (local address 127.0.98.3:45079)
0619 02:26:22.110873 (+ 1649us) negotiation.cc:107] Waiting for socket to connect
0619 02:26:22.110930 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 02:26:22.111963 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:26:22.112718 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:26:22.112727 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:26:22.113084 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:26:22.113771 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:22.113791 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.114953 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.114956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:22.116346 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:22.116356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.117082 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.117086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:22.118786 (+ 1700us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:22.119536 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:26:22.119557 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:26:22.121392 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:26:22.123576 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:26:22.123582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:26:22.123599 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:26:22.123924 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:26:22.124217 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:26:22.124220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:26:22.124221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:26:22.124440 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:26:22.124910 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:26:22.124917 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:26:22.125248 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 02:26:22.126331 (+ 1083us) client_negotiation.cc:240] Negotiation successful
0619 02:26:22.126698 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"thread_start_us":142,"threads_started":1}
I20260619 02:26:22.129877 12093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42831
I20260619 02:26:22.130121 12093 heartbeater.cc:464] Registering TS with master...
I20260619 02:26:22.130682 12093 heartbeater.cc:511] Master 127.0.98.62:42831 requested a full tablet report, sending...
I20260619 02:26:22.132750 11607 ts_manager.cc:194] Registered new tserver with Master: bdc637f980b44affb01602174042d0d3 (127.0.98.3:34905)
I20260619 02:26:22.133466 11607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45079
I20260619 02:26:22.144150 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:26:22.160645 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.146644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.0.98.62:42831)
0619 02:26:22.146762 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 02:26:22.146766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:26:22.147439 (+ 673us) server_negotiation.cc:407] Connection header received
0619 02:26:22.147641 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:26:22.147644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:26:22.147691 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:26:22.147740 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:26:22.148722 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.149612 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.150860 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.151200 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.151277 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:22.153187 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:26:22.153192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:26:22.153195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:26:22.153223 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:26:22.158686 (+ 5463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:26:22.159223 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:26:22.159228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:26:22.159230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:26:22.159320 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:26:22.159645 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:26:22.159649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:26:22.159651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:26:22.159873 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:26:22.159977 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:26:22.160332 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 02:26:22.160458 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:26:22.167866 11607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:26:22.175192 11607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:26:22.205858 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.192059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41303 (local address 127.0.0.1:43840)
0619 02:26:22.194232 (+ 2173us) negotiation.cc:107] Waiting for socket to connect
0619 02:26:22.194243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:26:22.194318 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:26:22.196653 (+ 2335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:26:22.196656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:26:22.196673 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:26:22.196767 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:22.196773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.202854 (+ 6081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.202858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:22.204048 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:22.204057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.205299 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.205302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:22.205419 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:22.205454 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 02:26:22.205579 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 02:26:22.205701 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2118,"thread_start_us":100,"threads_started":1}
I20260619 02:26:22.206418 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.192456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.0.98.1:41303)
0619 02:26:22.193593 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0619 02:26:22.193602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:26:22.194408 (+ 806us) server_negotiation.cc:407] Connection header received
0619 02:26:22.194489 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:26:22.194495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:26:22.194692 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:26:22.194828 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:26:22.196894 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.202184 (+ 5290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.204197 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.205007 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.205106 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:22.205242 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:26:22.206186 (+ 944us) server_negotiation.cc:299] Negotiation successful
0619 02:26:22.206288 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006,"thread_start_us":94,"threads_started":1}
I20260619 02:26:22.209825 11755 tablet_service.cc:1467] Processing CreateTablet for tablet 615e2d5f6b1f479fa201edee60d99962 (DEFAULT_TABLE table=test-table [id=f0834b0894f14c2c96485388e648b281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:26:22.211064 11755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615e2d5f6b1f479fa201edee60d99962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:26:22.211644 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.188740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34905 (local address 127.0.0.1:58664)
0619 02:26:22.193840 (+ 5100us) negotiation.cc:107] Waiting for socket to connect
0619 02:26:22.193893 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:26:22.194013 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:26:22.195033 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:26:22.195039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:26:22.195061 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:26:22.195208 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:22.195214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.200262 (+ 5048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.200265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:22.201679 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:22.201689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.211095 (+ 9406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.211098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:22.211331 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:22.211358 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 02:26:22.211441 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:26:22.211513 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4930,"thread_start_us":86,"threads_started":1}
I20260619 02:26:22.211761 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.189666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.0.98.3:34905)
0619 02:26:22.194336 (+ 4670us) server_negotiation.cc:206] Beginning negotiation
0619 02:26:22.194349 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:26:22.194373 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:26:22.194452 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:26:22.194458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:26:22.194687 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:26:22.194847 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:26:22.198387 (+ 3540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.200119 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.203111 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.204039 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.204157 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:22.204313 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 02:26:22.211487 (+ 7174us) server_negotiation.cc:299] Negotiation successful
0619 02:26:22.211607 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4554,"thread_start_us":84,"threads_started":1}
I20260619 02:26:22.216260 12026 tablet_service.cc:1467] Processing CreateTablet for tablet 615e2d5f6b1f479fa201edee60d99962 (DEFAULT_TABLE table=test-table [id=f0834b0894f14c2c96485388e648b281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:26:22.217417 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.191272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37947 (local address 127.0.0.1:37308)
0619 02:26:22.193986 (+ 2714us) negotiation.cc:107] Waiting for socket to connect
0619 02:26:22.194002 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:26:22.194107 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:26:22.198124 (+ 4017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:26:22.198127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:26:22.198146 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:26:22.198251 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:22.198258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.209671 (+ 11413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.209675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:22.210956 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:22.210966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.216990 (+ 6024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.216996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:22.217104 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:22.217127 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:26:22.217202 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:26:22.217278 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2643,"thread_start_us":81,"threads_started":1}
I20260619 02:26:22.217787 12026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615e2d5f6b1f479fa201edee60d99962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:26:22.217878 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.191659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.0.98.2:37947)
0619 02:26:22.196098 (+ 4439us) server_negotiation.cc:206] Beginning negotiation
0619 02:26:22.196118 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 02:26:22.196139 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:26:22.196219 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:26:22.196223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:26:22.196394 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:26:22.196527 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:26:22.201835 (+ 5308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.209503 (+ 7668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.212099 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.216674 (+ 4575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.216793 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:22.216927 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 02:26:22.217617 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 02:26:22.217730 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4334,"thread_start_us":111,"threads_started":1}
I20260619 02:26:22.220568 11893 tablet_service.cc:1467] Processing CreateTablet for tablet 615e2d5f6b1f479fa201edee60d99962 (DEFAULT_TABLE table=test-table [id=f0834b0894f14c2c96485388e648b281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:26:22.221728 11893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615e2d5f6b1f479fa201edee60d99962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:26:22.229555 12113 tablet_bootstrap.cc:492] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3: Bootstrap starting.
I20260619 02:26:22.229964 12114 tablet_bootstrap.cc:492] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3: Bootstrap starting.
I20260619 02:26:22.232146 12113 tablet_bootstrap.cc:654] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3: Neither blocks nor log segments found. Creating new log.
I20260619 02:26:22.232597 12114 tablet_bootstrap.cc:654] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3: Neither blocks nor log segments found. Creating new log.
I20260619 02:26:22.233232 12113 log.cc:824] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3: Log is configured to *not* fsync() on all Append() calls
I20260619 02:26:22.233381 12114 log.cc:824] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3: Log is configured to *not* fsync() on all Append() calls
I20260619 02:26:22.233860 12115 tablet_bootstrap.cc:492] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406: Bootstrap starting.
I20260619 02:26:22.235517 12113 tablet_bootstrap.cc:492] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3: No bootstrap required, opened a new log
I20260619 02:26:22.235698 12113 ts_tablet_manager.cc:1397] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:26:22.236402 12115 tablet_bootstrap.cc:654] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406: Neither blocks nor log segments found. Creating new log.
I20260619 02:26:22.237145 12115 log.cc:824] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406: Log is configured to *not* fsync() on all Append() calls
I20260619 02:26:22.239776 12115 tablet_bootstrap.cc:492] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406: No bootstrap required, opened a new log
I20260619 02:26:22.239995 12115 ts_tablet_manager.cc:1397] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:26:22.241811 12113 raft_consensus.cc:348] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc637f980b44affb01602174042d0d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "efa09dea8cf8416a8e7e50db390a99b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "e685b752cb2746df92754ae7368b7406" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37947 } }
I20260619 02:26:22.242170 12113 raft_consensus.cc:374] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:26:22.242254 12113 raft_consensus.cc:729] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa09dea8cf8416a8e7e50db390a99b3, State: Initialized, Role: FOLLOWER
Jun 19 02:26:22 dist-test-slave-xkrt krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:26:22.243732 12113 consensus_queue.cc:260] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc637f980b44affb01602174042d0d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "efa09dea8cf8416a8e7e50db390a99b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "e685b752cb2746df92754ae7368b7406" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37947 } }
I20260619 02:26:22.244443 12114 tablet_bootstrap.cc:492] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3: No bootstrap required, opened a new log
I20260619 02:26:22.244616 12114 ts_tablet_manager.cc:1397] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260619 02:26:22.244848 12113 ts_tablet_manager.cc:1428] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260619 02:26:22.245265 11823 heartbeater.cc:503] Master 127.0.98.62:42831 was elected leader, sending a full tablet report...
I20260619 02:26:22.246292 12115 raft_consensus.cc:348] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc637f980b44affb01602174042d0d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "efa09dea8cf8416a8e7e50db390a99b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "e685b752cb2746df92754ae7368b7406" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37947 } }
I20260619 02:26:22.246546 12115 raft_consensus.cc:374] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:26:22.246592 12115 raft_consensus.cc:729] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e685b752cb2746df92754ae7368b7406, State: Initialized, Role: FOLLOWER
I20260619 02:26:22.247032 12115 consensus_queue.cc:260] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc637f980b44affb01602174042d0d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "efa09dea8cf8416a8e7e50db390a99b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "e685b752cb2746df92754ae7368b7406" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37947 } }
I20260619 02:26:22.247933 12115 ts_tablet_manager.cc:1428] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260619 02:26:22.248277 11959 heartbeater.cc:503] Master 127.0.98.62:42831 was elected leader, sending a full tablet report...
I20260619 02:26:22.248344 12114 raft_consensus.cc:348] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc637f980b44affb01602174042d0d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "efa09dea8cf8416a8e7e50db390a99b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "e685b752cb2746df92754ae7368b7406" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37947 } }
I20260619 02:26:22.248582 12114 raft_consensus.cc:374] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:26:22.248639 12114 raft_consensus.cc:729] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdc637f980b44affb01602174042d0d3, State: Initialized, Role: FOLLOWER
I20260619 02:26:22.249023 12114 consensus_queue.cc:260] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc637f980b44affb01602174042d0d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "efa09dea8cf8416a8e7e50db390a99b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "e685b752cb2746df92754ae7368b7406" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37947 } }
I20260619 02:26:22.250001 12114 ts_tablet_manager.cc:1428] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:26:22.250311 12093 heartbeater.cc:503] Master 127.0.98.62:42831 was elected leader, sending a full tablet report...
I20260619 02:26:22.270390 12121 raft_consensus.cc:482] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:26:22.270823 12121 raft_consensus.cc:504] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc637f980b44affb01602174042d0d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "efa09dea8cf8416a8e7e50db390a99b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "e685b752cb2746df92754ae7368b7406" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37947 } }
I20260619 02:26:22.272372 12121 leader_election.cc:288] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efa09dea8cf8416a8e7e50db390a99b3 (127.0.98.1:41303), e685b752cb2746df92754ae7368b7406 (127.0.98.2:37947)
I20260619 02:26:22.277114 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.273002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59259 (local address 127.0.98.1:41303)
0619 02:26:22.273153 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:26:22.273156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:26:22.273169 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:26:22.273213 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:26:22.273216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:26:22.273286 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:26:22.273363 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:26:22.273809 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.274547 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.275792 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.276449 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.276550 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:22.276614 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:26:22.276948 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 02:26:22.277017 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:26:22.277107 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.272571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41303 (local address 127.0.98.3:59259)
0619 02:26:22.272693 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 02:26:22.272711 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:26:22.272804 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:26:22.273494 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:26:22.273496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:26:22.273513 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:26:22.273678 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:22.273686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.274689 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.274696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:22.275647 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:22.275656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.276673 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.276677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:22.276794 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:22.276824 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 02:26:22.276899 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:26:22.276975 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 02:26:22.277740 11763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615e2d5f6b1f479fa201edee60d99962" candidate_uuid: "bdc637f980b44affb01602174042d0d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa09dea8cf8416a8e7e50db390a99b3" is_pre_election: true
I20260619 02:26:22.278059 11763 raft_consensus.cc:2454] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdc637f980b44affb01602174042d0d3 in term 0.
I20260619 02:26:22.278600 11982 leader_election.cc:302] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdc637f980b44affb01602174042d0d3, efa09dea8cf8416a8e7e50db390a99b3; no voters:
I20260619 02:26:22.278867 12121 raft_consensus.cc:2781] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:26:22.278980 12121 raft_consensus.cc:482] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:26:22.279034 12121 raft_consensus.cc:3037] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:26:22.280223 12121 raft_consensus.cc:504] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc637f980b44affb01602174042d0d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "efa09dea8cf8416a8e7e50db390a99b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "e685b752cb2746df92754ae7368b7406" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37947 } }
I20260619 02:26:22.280282 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.272871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43365 (local address 127.0.98.2:37947)
0619 02:26:22.272993 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 02:26:22.272997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:26:22.273963 (+ 966us) server_negotiation.cc:407] Connection header received
0619 02:26:22.274153 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:26:22.274157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:26:22.274201 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:26:22.274248 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:26:22.274894 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.275655 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.278343 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.278961 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.279039 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:22.279096 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:26:22.280081 (+ 985us) server_negotiation.cc:299] Negotiation successful
0619 02:26:22.280158 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:26:22.280666 12121 leader_election.cc:288] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [CANDIDATE]: Term 1 election: Requested vote from peers efa09dea8cf8416a8e7e50db390a99b3 (127.0.98.1:41303), e685b752cb2746df92754ae7368b7406 (127.0.98.2:37947)
I20260619 02:26:22.280860 11763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615e2d5f6b1f479fa201edee60d99962" candidate_uuid: "bdc637f980b44affb01602174042d0d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa09dea8cf8416a8e7e50db390a99b3"
I20260619 02:26:22.281008 11763 raft_consensus.cc:3037] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:26:22.281570 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.272574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37947 (local address 127.0.98.3:43365)
0619 02:26:22.273700 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0619 02:26:22.273716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:26:22.274034 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:26:22.274378 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:26:22.274381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:26:22.274397 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:26:22.274761 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:22.274773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.276003 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.276006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:22.278214 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:26:22.278225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.279841 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.279845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:26:22.279942 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:22.279965 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:26:22.281008 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0619 02:26:22.281085 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":78,"thread_start_us":119,"threads_started":1}
I20260619 02:26:22.282259 11912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615e2d5f6b1f479fa201edee60d99962" candidate_uuid: "bdc637f980b44affb01602174042d0d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e685b752cb2746df92754ae7368b7406" is_pre_election: true
I20260619 02:26:22.282297 11913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615e2d5f6b1f479fa201edee60d99962" candidate_uuid: "bdc637f980b44affb01602174042d0d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e685b752cb2746df92754ae7368b7406"
I20260619 02:26:22.282337 11763 raft_consensus.cc:2454] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdc637f980b44affb01602174042d0d3 in term 1.
I20260619 02:26:22.282526 11912 raft_consensus.cc:2454] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdc637f980b44affb01602174042d0d3 in term 0.
I20260619 02:26:22.282619 11982 leader_election.cc:302] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdc637f980b44affb01602174042d0d3, efa09dea8cf8416a8e7e50db390a99b3; no voters:
I20260619 02:26:22.282788 12121 raft_consensus.cc:2781] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:26:22.283052 12121 raft_consensus.cc:686] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [term 1 LEADER]: Becoming Leader. State: Replica: bdc637f980b44affb01602174042d0d3, State: Running, Role: LEADER
I20260619 02:26:22.283380 12121 consensus_queue.cc:237] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc637f980b44affb01602174042d0d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "efa09dea8cf8416a8e7e50db390a99b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "e685b752cb2746df92754ae7368b7406" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37947 } }
I20260619 02:26:22.287053 11609 catalog_manager.cc:5495] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdc637f980b44affb01602174042d0d3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bdc637f980b44affb01602174042d0d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc637f980b44affb01602174042d0d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efa09dea8cf8416a8e7e50db390a99b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e685b752cb2746df92754ae7368b7406" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37947 } health_report { overall_health: UNKNOWN } } }
I20260619 02:26:22.326323 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.321221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.0.98.3:34905)
0619 02:26:22.321347 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:26:22.321351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:26:22.321365 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:26:22.321455 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:26:22.321458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:26:22.321528 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:26:22.321673 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:26:22.322082 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.322835 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.324090 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:26:22.324399 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:26:22.324499 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:26:22.324929 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:26:22.325495 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:26:22.325697 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 02:26:22.326116 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 02:26:22.326181 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:26:22.346326 11913 raft_consensus.cc:3037] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:26:22.347746 11913 raft_consensus.cc:1261] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406 [term 1 FOLLOWER]: Refusing update from remote peer bdc637f980b44affb01602174042d0d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:26:22.348402 12121 consensus_queue.cc:1035] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e685b752cb2746df92754ae7368b7406" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:26:22.348912 11763 raft_consensus.cc:1261] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3 [term 1 FOLLOWER]: Refusing update from remote peer bdc637f980b44affb01602174042d0d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:26:22.349783 12121 consensus_queue.cc:1035] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efa09dea8cf8416a8e7e50db390a99b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:26:22.357623 12094 tablet.cc:2369] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:26:22.366777 11824 tablet.cc:2369] T 615e2d5f6b1f479fa201edee60d99962 P efa09dea8cf8416a8e7e50db390a99b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:26:22.390296 12128 mvcc.cc:205] Tried to move back new op lower bound from 7298400183680880640 to 7298400183435591680. Current Snapshot: MvccSnapshot[applied={T|T < 7298400183680880640}]
I20260619 02:26:22.391191 12130 mvcc.cc:205] Tried to move back new op lower bound from 7298400183680880640 to 7298400183435591680. Current Snapshot: MvccSnapshot[applied={T|T < 7298400183680880640}]
I20260619 02:26:22.400729 11609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:26:22.400914 11609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:26:22.403800 11609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb03991c6ad648da910bfc1432c62d7c: Sending DeleteTablet for 3 replicas of tablet 615e2d5f6b1f479fa201edee60d99962
I20260619 02:26:22.404701 12026 tablet_service.cc:1514] Processing DeleteTablet for tablet 615e2d5f6b1f479fa201edee60d99962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:26:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58664
I20260619 02:26:22.404767 11755 tablet_service.cc:1514] Processing DeleteTablet for tablet 615e2d5f6b1f479fa201edee60d99962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:26:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43840
I20260619 02:26:22.405875 11893 tablet_service.cc:1514] Processing DeleteTablet for tablet 615e2d5f6b1f479fa201edee60d99962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:26:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37308
I20260619 02:26:22.407017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 11690
I20260619 02:26:22.418957 12138 tablet_replica.cc:330] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406: stopping tablet replica
I20260619 02:26:22.419432 12138 raft_consensus.cc:2229] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:26:22.419970 12138 raft_consensus.cc:2258] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:26:22.421721 12136 tablet_replica.cc:330] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3: stopping tablet replica
I20260619 02:26:22.421777 12138 ts_tablet_manager.cc:1905] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:26:22.422096 12136 raft_consensus.cc:2229] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:26:22.422535 12136 raft_consensus.cc:2258] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:26:22.431092 12136 ts_tablet_manager.cc:1905] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:26:22.431968 12138 ts_tablet_manager.cc:1918] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:26:22.432077 12138 log.cc:1196] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/615e2d5f6b1f479fa201edee60d99962
I20260619 02:26:22.432397 12138 ts_tablet_manager.cc:1939] T 615e2d5f6b1f479fa201edee60d99962 P e685b752cb2746df92754ae7368b7406: Deleting consensus metadata
I20260619 02:26:22.433761 11596 catalog_manager.cc:4841] TS e685b752cb2746df92754ae7368b7406 (127.0.98.2:37947): tablet 615e2d5f6b1f479fa201edee60d99962 (table test-table [id=f0834b0894f14c2c96485388e648b281]) successfully deleted
W20260619 02:26:22.434314 11595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41303 (error 108)
I20260619 02:26:22.436403 12136 ts_tablet_manager.cc:1918] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:26:22.436532 12136 log.cc:1196] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/615e2d5f6b1f479fa201edee60d99962
I20260619 02:26:22.436830 12136 ts_tablet_manager.cc:1939] T 615e2d5f6b1f479fa201edee60d99962 P bdc637f980b44affb01602174042d0d3: Deleting consensus metadata
I20260619 02:26:22.437942 11595 catalog_manager.cc:4841] TS bdc637f980b44affb01602174042d0d3 (127.0.98.3:34905): tablet 615e2d5f6b1f479fa201edee60d99962 (table test-table [id=f0834b0894f14c2c96485388e648b281]) successfully deleted
I20260619 02:26:22.438237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 11829
I20260619 02:26:22.446844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 11964
I20260619 02:26:22.447048 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:26:22.446705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41303 (local address 127.0.0.1:43846)
0619 02:26:22.446858 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 02:26:22.446942 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 02:26:22.447227 11595 catalog_manager.cc:4584] TS efa09dea8cf8416a8e7e50db390a99b3 (127.0.98.1:41303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 615e2d5f6b1f479fa201edee60d99962: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41303: connect: Connection refused (error 111)
I20260619 02:26:22.455104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 11577
I20260619 02:26:22.460741 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 02:26:22.805246 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 02:26:22.805368 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 02:26:23.150185 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 02:26:23.150 UTC [11163] LOG: received smart shutdown request
2026-06-19 02:26:23.153 UTC [11163] LOG: background worker "logical replication launcher" (PID 11172) exited with exit code 1
2026-06-19 02:26:23.153 UTC [11167] LOG: shutting down
2026-06-19 02:26:23.321 UTC [11163] LOG: database system is shut down
2026-06-19T02:26:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44510 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:26:23 dist-test-slave-xkrt krb5kdc[12148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:26:23 dist-test-slave-xkrt krb5kdc[12148](info): set up 2 sockets
Jun 19 02:26:23 dist-test-slave-xkrt krb5kdc[12148](info): commencing operation
krb5kdc: starting...
W20260619 02:26:24.505767 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:26:24 dist-test-slave-xkrt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781835984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:26:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:26:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 02:26:24.675712 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjWwzrN/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/postgres -l logfile start
2026-06-19 02:26:27.209 UTC [12178] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 02:26:27.209 UTC [12178] LOG: listening on IPv4 address "127.0.98.20", port 54399
2026-06-19 02:26:27.213 UTC [12178] LOG: listening on Unix socket "/tmp/.s.PGSQL.54399"
2026-06-19 02:26:27.247 UTC [12181] LOG: database system was shut down at 2026-06-19 02:26:25 UTC
2026-06-19 02:26:27.251 UTC [12178] LOG: database system is ready to accept connections
I20260619 02:26:28.205401 392 mini_postgres.cc:96] Postgres bound to 54399
2026-06-19 02:26:28.212 UTC [12189] FATAL: database "slave" does not exist
127.0.98.20:54399 - accepting connections
I20260619 02:26:28.212873 392 mini_ranger.cc:161] Starting Ranger...
I20260619 02:26:28.228354 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 02:26:28.698843 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 02:26:28.698926 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-admin
2026-06-19 02:26:28,971 [I] DB FLAVOR :POSTGRES
2026-06-19 02:26:28,972 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:26:28,972 [I] Checking connection..
2026-06-19 02:26:28,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 02:26:29,230 [I] Checking connection passed.
2026-06-19 02:26:29,231 [I] --------- Verifying version history table ---------
2026-06-19 02:26:29,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:26:29,485 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:26:29,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:26:29,710 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:26:29,710 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 02:26:29,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 02:26:29.955 UTC [12253] WARNING: there is no transaction in progress
2026-06-19 02:26:29,961 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 02:26:29,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:26:30,192 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 02:26:30,192 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:26:30,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 02:26:30,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 02:26:30,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xkrt.c.gcp-upstream.internal','N') ;"
2026-06-19 02:26:30,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:26:31,192 [I] Table x_portal_user does not exist in database ranger
2026-06-19 02:26:31,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:26:31,448 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 02:26:31,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:26:31,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:26:31,893 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 02:26:31,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 02:26:32.789 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:32.805 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:32.821 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:32.835 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:32.850 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:32.866 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:32.918 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:32.924 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:32.938 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:32.947 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:32.957 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:32.969 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:32.979 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:32.986 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:32.993 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:33.000 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 02:26:33,632 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 02:26:33,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 02:26:33,882 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 02:26:33,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:26:34,108 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 02:26:34,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:26:34,363 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 02:26:34,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:26:34,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:26:34,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xkrt.c.gcp-upstream.internal';"
2026-06-19 02:26:35,100 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 02:26:35,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 02:26:35,337 [I] DB_PATCHES have already been applied
I20260619 02:26:35.342403 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjWwzrN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 02:26:35.342459 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 02:26:35.345752 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:55877
Jun 19, 2026 2:26:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:26:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:26:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:26:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:26:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:26:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:26:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 02:26:36.462571 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.250s user 0.000s sys 0.009s
Jun 19, 2026 2:26:56 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:26:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 2:26:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:26:56 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:26:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:26:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:26:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:26:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:26:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:26:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:26:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:26:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:26:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:26:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:26:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:26:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:26:56 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:26:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 02:26:58.109714 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 02:26:58 dist-test-slave-xkrt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836018, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 02:26:58.190047 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 02:26:58.199329 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 02:26:59.022364 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 02:26:59.022460 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-kms
I20260619 02:26:59.022480 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54399
2026-06-19 02:26:59,184 [I] DB FLAVOR :POSTGRES
2026-06-19 02:26:59,185 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:26:59,185 [I] Checking connection
2026-06-19 02:26:59,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 02:26:59,415 [I] connection success
2026-06-19 02:26:59,416 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 02:26:59,416 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 02:26:59,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 02:26:59,648 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 02:26:59,649 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:26:59,649 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 02:26:59,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54399/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 02:26:59,890 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 02:27:00.335446 392 mini_ranger_kms.cc:310] Created kms service
I20260619 02:27:00.511793 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 02:27:00.555054 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 02:27:00 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836018, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:27:00.934352 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 02:27:00.934494 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjWwzrN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjWwzrN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 02:27:00.934513 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 02:27:00.938387 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12541
I20260619 02:27:00.938468 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:59721
Jun 19, 2026 2:27:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:27:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:27:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:27:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:27:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:27:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 02:27:03.265273 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.075s user 0.009s sys 0.137s
I20260619 02:27:03.265398 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 02:27:03.265465 392 mini_ranger_kms.cc:401] 127.0.98.20:59721/kms/v1/keys
Jun 19 02:27:03 dist-test-slave-xkrt krb5kdc[12148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781836023, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 02:27:04 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836023, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 02:27:04 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836023, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 02:27:06 dist-test-slave-xkrt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:27:06.617646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39375
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39375
--ranger_config_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59721/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:06.726055 12592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:06.726294 12592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:06.726332 12592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:06.730474 12592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:27:06.730533 12592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:06.730553 12592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:06.730572 12592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:27:06.730612 12592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:27:06.736267 12592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:59721/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39375
--ranger_config_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:06.737195 12592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:06.738205 12592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:06.746520 12598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:06.746549 12592 server_base.cc:1034] running on GCE node
W20260619 02:27:06.746549 12597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:06.746557 12600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:06.747337 12592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:06.748353 12592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:06.749516 12592 hybrid_clock.cc:648] HybridClock initialized: now 1781836026749499 us; error 29 us; skew 500 ppm
Jun 19 02:27:06 dist-test-slave-xkrt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:06.752187 12592 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:27:06.753063 12592 webserver.cc:460] Webserver started at http://127.0.98.62:39915/ using document root <none> and password file <none>
I20260619 02:27:06.753647 12592 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:06.753705 12592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:06.753931 12592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:27:06 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:27:06.799763 12592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "ab46bec829004c52b815af7df87222f6"
format_stamp: "Formatted at 2026-06-19 02:27:06 on dist-test-slave-xkrt"
server_key: "cfcbe290678c871b5a14e83724d4c5d9"
server_key_iv: "eeb4eec08c612a0b1c440358bc51e0a2"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:06.800393 12592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab46bec829004c52b815af7df87222f6"
format_stamp: "Formatted at 2026-06-19 02:27:06 on dist-test-slave-xkrt"
server_key: "cfcbe290678c871b5a14e83724d4c5d9"
server_key_iv: "eeb4eec08c612a0b1c440358bc51e0a2"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:06.804580 12592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:27:06.818130 12607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:06.819577 12592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:27:06.819720 12592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "ab46bec829004c52b815af7df87222f6"
format_stamp: "Formatted at 2026-06-19 02:27:06 on dist-test-slave-xkrt"
server_key: "cfcbe290678c871b5a14e83724d4c5d9"
server_key_iv: "eeb4eec08c612a0b1c440358bc51e0a2"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:06.819871 12592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:06.856654 12592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:06.859238 12592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:06.859445 12592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:06.866488 12592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39375
I20260619 02:27:06.866503 12659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39375 every 8 connection(s)
I20260619 02:27:06.867408 12592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:27:06.870041 12660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:06.874424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 12592
I20260619 02:27:06.874594 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:27:06.874922 392 external_mini_cluster.cc:1428] Setting key e5e1c8ba4da6ad31703ec21d0efeeff3
I20260619 02:27:06.875383 12660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6: Bootstrap starting.
I20260619 02:27:06.877684 12660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:06.878497 12660 log.cc:824] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:06.880395 12660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6: No bootstrap required, opened a new log
Jun 19 02:27:06 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:06.884778 12660 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab46bec829004c52b815af7df87222f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39375 } }
I20260619 02:27:06.885015 12660 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:06.885072 12660 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab46bec829004c52b815af7df87222f6, State: Initialized, Role: FOLLOWER
I20260619 02:27:06.885504 12660 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab46bec829004c52b815af7df87222f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39375 } }
I20260619 02:27:06.885680 12660 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:27:06.885783 12660 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:27:06.885915 12660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:06.886706 12660 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab46bec829004c52b815af7df87222f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39375 } }
I20260619 02:27:06.887032 12660 leader_election.cc:302] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab46bec829004c52b815af7df87222f6; no voters:
I20260619 02:27:06.887334 12660 leader_election.cc:288] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:27:06.887452 12665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:27:06.887759 12665 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 1 LEADER]: Becoming Leader. State: Replica: ab46bec829004c52b815af7df87222f6, State: Running, Role: LEADER
I20260619 02:27:06.888123 12665 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab46bec829004c52b815af7df87222f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39375 } }
I20260619 02:27:06.888640 12660 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:27:06.889925 12666 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab46bec829004c52b815af7df87222f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab46bec829004c52b815af7df87222f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39375 } } }
I20260619 02:27:06.890055 12666 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:06.890501 12676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:27:06.890450 12667 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab46bec829004c52b815af7df87222f6. Latest consensus state: current_term: 1 leader_uuid: "ab46bec829004c52b815af7df87222f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab46bec829004c52b815af7df87222f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39375 } } }
I20260619 02:27:06.890669 12667 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:06.890815 12663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:06.876441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.0.98.62:39375)
0619 02:27:06.876898 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:06.876913 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:06.876946 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:27:06.877604 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:06.877622 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:06.877981 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:06.878263 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:06.879162 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:06.880343 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:06.881477 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:06.882046 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:06.882170 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:06.885078 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:06.885088 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:06.885105 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:06.885142 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:06.887290 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:06.887825 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:06.887831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:06.887837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:06.887956 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:06.888247 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:06.888251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:06.888253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:06.888888 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:06.889086 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:06.889392 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 02:27:06.889753 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":169,"threads_started":1}
I20260619 02:27:06.890846 12660 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:27:06.893525 12676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:27:06.902413 12676 catalog_manager.cc:1347] Generated new cluster ID: 38196580bbef4269a88bc9aeaf9293f2
I20260619 02:27:06.902477 12676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:27:06.915596 12676 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:27:06.916874 12676 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:27:06.927875 12676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6: Generated new TSK 0
I20260619 02:27:06.928617 12676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:27:07.589847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59721/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39375
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:07.753151 12705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:07.753598 12705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:07.753724 12705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:07.758821 12705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:07.758910 12705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:07.759032 12705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:27:07.767414 12705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:59721/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:07.769075 12705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:07.770515 12705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:07.784514 12713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:07.784988 12715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:07.786144 12705 server_base.cc:1034] running on GCE node
W20260619 02:27:07.788013 12712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:07.788514 12705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:07.789266 12705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:07.790447 12705 hybrid_clock.cc:648] HybridClock initialized: now 1781836027790420 us; error 47 us; skew 500 ppm
Jun 19 02:27:07 dist-test-slave-xkrt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:07.793272 12705 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:27:07.794384 12705 webserver.cc:460] Webserver started at http://127.0.98.1:43487/ using document root <none> and password file <none>
I20260619 02:27:07.794943 12705 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:07.795032 12705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:07.795264 12705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:27:07 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:27:07.811873 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "5c7574c570344c3197342f24bed4120b"
format_stamp: "Formatted at 2026-06-19 02:27:07 on dist-test-slave-xkrt"
server_key: "c994b0cf55d0b4d57213fce2d31b2a6d"
server_key_iv: "f0ee9f50a451c6419728727192009fd9"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:07.812610 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5c7574c570344c3197342f24bed4120b"
format_stamp: "Formatted at 2026-06-19 02:27:07 on dist-test-slave-xkrt"
server_key: "c994b0cf55d0b4d57213fce2d31b2a6d"
server_key_iv: "f0ee9f50a451c6419728727192009fd9"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:07.826193 12705 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.001s sys 0.006s
I20260619 02:27:07.847309 12724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:07.850622 12705 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260619 02:27:07.850827 12705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "5c7574c570344c3197342f24bed4120b"
format_stamp: "Formatted at 2026-06-19 02:27:07 on dist-test-slave-xkrt"
server_key: "c994b0cf55d0b4d57213fce2d31b2a6d"
server_key_iv: "f0ee9f50a451c6419728727192009fd9"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:07.851015 12705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:07.876266 12705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:07.879536 12705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:07.879798 12705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:07.880460 12705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:07.881313 12705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:07.881381 12705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:07.881464 12705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:07.881518 12705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:27:07 dist-test-slave-xkrt krb5kdc[12148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781836027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:07.892144 12705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41919
I20260619 02:27:07.892289 12837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41919 every 8 connection(s)
I20260619 02:27:07.893076 12705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:27:07.898623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 12705
I20260619 02:27:07.898741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:27:07.899057 392 external_mini_cluster.cc:1428] Setting key e3be9ae57ffa9eff5839d6c8f9310047
Jun 19 02:27:07 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:07.909792 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:07.894841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56429 (local address 127.0.98.62:39375)
0619 02:27:07.895102 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:07.895109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:07.895962 (+ 853us) server_negotiation.cc:407] Connection header received
0619 02:27:07.896910 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:07.896914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:07.896973 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:07.897061 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:07.898345 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:07.899187 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:07.901886 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:07.902222 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:07.902298 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:07.905413 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:07.905418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:07.905421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:07.905447 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:07.907356 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:07.907973 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:07.907977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:07.907979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:07.908063 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:07.908453 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:07.908457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:07.908459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:07.908671 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:07.908755 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:07.909426 (+ 671us) server_negotiation.cc:299] Negotiation successful
0619 02:27:07.909583 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260619 02:27:07.911598 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:07.895295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39375 (local address 127.0.98.1:56429)
0619 02:27:07.895821 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:07.895868 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:07.896716 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:07.897218 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:07.897233 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:07.897479 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:07.898168 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:07.898190 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:07.900481 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:07.900485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:07.901752 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:07.901763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:07.902509 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:07.902512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:07.902709 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:07.903366 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:07.903386 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:07.905226 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:07.907487 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:07.907492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:07.907508 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:07.907854 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:07.908174 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:07.908177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:07.908179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:07.908345 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:07.908835 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:07.908841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:07.909150 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 02:27:07.910259 (+ 1109us) client_negotiation.cc:240] Negotiation successful
0619 02:27:07.910579 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":135,"threads_started":1}
I20260619 02:27:07.913184 12838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39375
I20260619 02:27:07.913444 12838 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:07.914017 12838 heartbeater.cc:511] Master 127.0.98.62:39375 requested a full tablet report, sending...
I20260619 02:27:07.915689 12624 ts_manager.cc:194] Registered new tserver with Master: 5c7574c570344c3197342f24bed4120b (127.0.98.1:41919)
I20260619 02:27:07.917198 12624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56429
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:27:07.948771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59721/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39375
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:08.109737 12844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:08.110164 12844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:08.110296 12844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:08.116716 12844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:08.116864 12844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:08.117009 12844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:27:08.125034 12844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:59721/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:08.126688 12844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:08.128109 12844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:08.140273 12850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:08.140391 12851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:08.140282 12853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:08.140710 12844 server_base.cc:1034] running on GCE node
I20260619 02:27:08.141114 12844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:08.141805 12844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:08.142993 12844 hybrid_clock.cc:648] HybridClock initialized: now 1781836028142979 us; error 33 us; skew 500 ppm
Jun 19 02:27:08 dist-test-slave-xkrt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:08.145885 12844 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:27:08.146836 12844 webserver.cc:460] Webserver started at http://127.0.98.2:41347/ using document root <none> and password file <none>
I20260619 02:27:08.147382 12844 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:08.147469 12844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:08.147681 12844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:27:08 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:27:08.163741 12844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "9c87dd22e022460fb17b8862fe49ddf9"
format_stamp: "Formatted at 2026-06-19 02:27:08 on dist-test-slave-xkrt"
server_key: "a51d6c798d0cbe6d939a7ba6e10c4441"
server_key_iv: "c458b43e3a9a8fc4d36531eaf2ebac1d"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:08.164860 12844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9c87dd22e022460fb17b8862fe49ddf9"
format_stamp: "Formatted at 2026-06-19 02:27:08 on dist-test-slave-xkrt"
server_key: "a51d6c798d0cbe6d939a7ba6e10c4441"
server_key_iv: "c458b43e3a9a8fc4d36531eaf2ebac1d"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:08.171429 12844 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 02:27:08.188695 12860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:08.190727 12844 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:27:08.190868 12844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "9c87dd22e022460fb17b8862fe49ddf9"
format_stamp: "Formatted at 2026-06-19 02:27:08 on dist-test-slave-xkrt"
server_key: "a51d6c798d0cbe6d939a7ba6e10c4441"
server_key_iv: "c458b43e3a9a8fc4d36531eaf2ebac1d"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:08.191028 12844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:08.214586 12844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:08.218235 12844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:08.218425 12844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:08.219141 12844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:08.220150 12844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:08.220222 12844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:08.220278 12844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:08.220346 12844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 02:27:08.235109 12844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35725
I20260619 02:27:08.235273 12973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35725 every 8 connection(s)
I20260619 02:27:08.236359 12844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:27:08.242733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 12844
I20260619 02:27:08.242861 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:27:08.243160 392 external_mini_cluster.cc:1428] Setting key 8f374653a7269447b9b0518ccb266e6b
Jun 19 02:27:08 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:08.254825 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:08.238404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46543 (local address 127.0.98.62:39375)
0619 02:27:08.238567 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:08.238572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:08.239598 (+ 1026us) server_negotiation.cc:407] Connection header received
0619 02:27:08.240838 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:08.240842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:08.240898 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:08.240957 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:08.243857 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:08.244713 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:08.246455 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:08.246803 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:08.246884 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:08.250363 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:08.250367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:08.250370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:08.250399 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:08.252333 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:08.252975 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:08.252979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:08.252981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:08.253069 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:08.253468 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:08.253471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:08.253473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:08.253718 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:08.253816 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:08.254490 (+ 674us) server_negotiation.cc:299] Negotiation successful
0619 02:27:08.254619 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:27:08.256582 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:08.238876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39375 (local address 127.0.98.2:46543)
0619 02:27:08.239438 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:08.239496 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:08.240587 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:08.241375 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:08.241385 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:08.241763 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:08.242419 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:08.242439 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:08.244875 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:08.244879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:08.246314 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:08.246325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:08.247084 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:08.247088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:08.247312 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:08.248047 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:08.248068 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:08.250173 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:08.252474 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:08.252481 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:08.252497 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:08.252852 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:08.253179 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:08.253182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:08.253184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:08.253354 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:08.253885 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:08.253892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:08.254211 (+ 319us) client_negotiation.cc:769] Sending connection context
0619 02:27:08.255052 (+ 841us) client_negotiation.cc:240] Negotiation successful
0619 02:27:08.255390 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":147,"threads_started":1}
I20260619 02:27:08.259509 12974 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39375
I20260619 02:27:08.259789 12974 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:08.260373 12974 heartbeater.cc:511] Master 127.0.98.62:39375 requested a full tablet report, sending...
I20260619 02:27:08.261602 12624 ts_manager.cc:194] Registered new tserver with Master: 9c87dd22e022460fb17b8862fe49ddf9 (127.0.98.2:35725)
I20260619 02:27:08.262418 12624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46543
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:27:08.288225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59721/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39375
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:27:08 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 02:27:08.442597 12979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:08.442956 12979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:08.443015 12979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:08.448897 12979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:08.448977 12979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:08.449059 12979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:27:08.456566 12979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:59721/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:08.458112 12979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:08.459393 12979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:08.471457 12985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:08.471464 12984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:08.471954 12987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:08.472819 12979 server_base.cc:1034] running on GCE node
I20260619 02:27:08.473434 12979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:08.474174 12979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:08.475364 12979 hybrid_clock.cc:648] HybridClock initialized: now 1781836028475338 us; error 43 us; skew 500 ppm
Jun 19 02:27:08 dist-test-slave-xkrt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:08.478878 12979 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:27:08.479960 12979 webserver.cc:460] Webserver started at http://127.0.98.3:40861/ using document root <none> and password file <none>
I20260619 02:27:08.480677 12979 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:08.480775 12979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:08.481083 12979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:27:08 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:27:08.500931 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "c51407af537a484b8c1a8a4cfa1519df"
format_stamp: "Formatted at 2026-06-19 02:27:08 on dist-test-slave-xkrt"
server_key: "82fcd1bfde914c1b8b33f22f58ffe5e3"
server_key_iv: "59fc999edc10aa4d8bc940999e0b6898"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:08.501724 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c51407af537a484b8c1a8a4cfa1519df"
format_stamp: "Formatted at 2026-06-19 02:27:08 on dist-test-slave-xkrt"
server_key: "82fcd1bfde914c1b8b33f22f58ffe5e3"
server_key_iv: "59fc999edc10aa4d8bc940999e0b6898"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:08.507747 12979 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 02:27:08.533962 12994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:08.535856 12979 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
I20260619 02:27:08.536006 12979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "c51407af537a484b8c1a8a4cfa1519df"
format_stamp: "Formatted at 2026-06-19 02:27:08 on dist-test-slave-xkrt"
server_key: "82fcd1bfde914c1b8b33f22f58ffe5e3"
server_key_iv: "59fc999edc10aa4d8bc940999e0b6898"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:08.536159 12979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:08.567085 12979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:08.571350 12979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:08.571557 12979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:08.572167 12979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:08.573196 12979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:08.573285 12979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:08.573335 12979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:08.573364 12979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:08.585810 12979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44825
I20260619 02:27:08.586943 12979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:27:08.590183 13107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44825 every 8 connection(s)
I20260619 02:27:08.595667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 12979
I20260619 02:27:08.595780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:27:08.596073 392 external_mini_cluster.cc:1428] Setting key a8d6fb95f4bb6631a119d80572d5cfc9
Jun 19 02:27:08 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:08.608177 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:08.589941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53573 (local address 127.0.98.62:39375)
0619 02:27:08.590066 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:08.590071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:08.591694 (+ 1623us) server_negotiation.cc:407] Connection header received
0619 02:27:08.592917 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:08.592921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:08.592975 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:08.593063 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:08.594564 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:08.595357 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:08.600233 (+ 4876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:08.600579 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:08.600676 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:08.603793 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:08.603797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:08.603800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:08.603828 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:08.605790 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:08.606408 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:08.606412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:08.606414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:08.606499 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:08.606891 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:08.606895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:08.606897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:08.607113 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:08.607198 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:08.607893 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 02:27:08.608016 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:27:08.610098 13110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:08.590489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39375 (local address 127.0.98.3:53573)
0619 02:27:08.591512 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:08.591571 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:08.592649 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:08.593366 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:08.593375 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:08.593735 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:08.594385 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:08.594403 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:08.596073 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:08.596077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:08.597829 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:08.597848 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:08.600888 (+ 3040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:08.600893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:08.601101 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:08.601830 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:08.601852 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:08.603561 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:08.605927 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:08.605933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:08.605949 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:08.606288 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:08.606606 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:08.606609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:08.606611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:08.606777 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:08.607277 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:08.607283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:08.607599 (+ 316us) client_negotiation.cc:769] Sending connection context
0619 02:27:08.608706 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0619 02:27:08.609030 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":123,"threads_started":1}
I20260619 02:27:08.611199 13108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39375
I20260619 02:27:08.611431 13108 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:08.611907 13108 heartbeater.cc:511] Master 127.0.98.62:39375 requested a full tablet report, sending...
I20260619 02:27:08.612874 12624 ts_manager.cc:194] Registered new tserver with Master: c51407af537a484b8c1a8a4cfa1519df (127.0.98.3:44825)
I20260619 02:27:08.613485 12624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53573
I20260619 02:27:08.621603 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:27:08.622386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 12705
Jun 19 02:27:08 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:27:08.643916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 12844
I20260619 02:27:08.652042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 12979
I20260619 02:27:08.658589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 12592
I20260619 02:27:08.666030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39375
--webserver_interface=127.0.98.62
--webserver_port=39915
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39375
--ranger_config_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59721/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59721/kms with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:08.788838 13112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:08.789100 13112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:08.789139 13112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:08.793061 13112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:27:08.793113 13112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:08.793133 13112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:08.793149 13112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:27:08.793164 13112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:27:08.798547 13112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59721/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39375
--ranger_config_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:08.799504 13112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:08.800514 13112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:08.808946 13121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:08.808965 13119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:08.808952 13118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:08.809448 13112 server_base.cc:1034] running on GCE node
I20260619 02:27:08.809998 13112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:08.811060 13112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:08.812235 13112 hybrid_clock.cc:648] HybridClock initialized: now 1781836028812221 us; error 31 us; skew 500 ppm
Jun 19 02:27:08 dist-test-slave-xkrt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:08.814952 13112 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:27:08.815876 13112 webserver.cc:460] Webserver started at http://127.0.98.62:39915/ using document root <none> and password file <none>
I20260619 02:27:08.816385 13112 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:08.816447 13112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:27:08 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:27:08.896325 13112 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:27:08.898572 13129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:08.899860 13112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:27:08.899995 13112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "ab46bec829004c52b815af7df87222f6"
format_stamp: "Formatted at 2026-06-19 02:27:06 on dist-test-slave-xkrt"
server_key: "cfcbe290678c871b5a14e83724d4c5d9"
server_key_iv: "eeb4eec08c612a0b1c440358bc51e0a2"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:08.900523 13112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:08.919656 13112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:08.922561 13112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:08.922765 13112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:08.930744 13112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39375
I20260619 02:27:08.930745 13181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39375 every 8 connection(s)
I20260619 02:27:08.931823 13112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:27:08.935957 13182 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:27:08.937520 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6: Bootstrap starting.
I20260619 02:27:08.938644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 13112
I20260619 02:27:08.939123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59721/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41919
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39375
--webserver_port=43487
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59721/kms with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:27:08.959507 13182 log.cc:824] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:08.967898 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:27:08.968343 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6: Bootstrap complete.
I20260619 02:27:08.973803 13182 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab46bec829004c52b815af7df87222f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39375 } }
I20260619 02:27:08.974320 13182 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab46bec829004c52b815af7df87222f6, State: Initialized, Role: FOLLOWER
I20260619 02:27:08.974910 13182 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab46bec829004c52b815af7df87222f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39375 } }
I20260619 02:27:08.975055 13182 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:27:08.975142 13182 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:27:08.975242 13182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:27:08.977636 13182 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab46bec829004c52b815af7df87222f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39375 } }
I20260619 02:27:08.977985 13182 leader_election.cc:302] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab46bec829004c52b815af7df87222f6; no voters:
I20260619 02:27:08.978317 13182 leader_election.cc:288] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:27:08.978483 13186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:27:08.978744 13186 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [term 2 LEADER]: Becoming Leader. State: Replica: ab46bec829004c52b815af7df87222f6, State: Running, Role: LEADER
I20260619 02:27:08.979099 13186 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab46bec829004c52b815af7df87222f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39375 } }
I20260619 02:27:08.979557 13182 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:27:08.980119 13186 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ab46bec829004c52b815af7df87222f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab46bec829004c52b815af7df87222f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39375 } } }
I20260619 02:27:08.980310 13186 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:08.980551 13186 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab46bec829004c52b815af7df87222f6. Latest consensus state: current_term: 2 leader_uuid: "ab46bec829004c52b815af7df87222f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab46bec829004c52b815af7df87222f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39375 } } }
I20260619 02:27:08.980674 13186 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:08.980947 13193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:27:08.984179 13193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:27:08.984678 13182 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:27:08.984903 13193 catalog_manager.cc:1259] Loaded cluster ID: 38196580bbef4269a88bc9aeaf9293f2
I20260619 02:27:08.985000 13193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:27:08.987394 13193 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:27:08.988137 13193 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6: Loaded TSK: 0
I20260619 02:27:08.989326 13193 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:27:09.092473 13184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:09.092818 13184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:09.092882 13184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:09.099238 13184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:09.099386 13184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:09.099519 13184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:27:09.106987 13184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59721/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43487
--tserver_master_addrs=127.0.98.62:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:09.108191 13184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:09.109390 13184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:09.121315 13222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:09.121505 13184 server_base.cc:1034] running on GCE node
W20260619 02:27:09.122388 13220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:09.122601 13219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:09.123098 13184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:09.123802 13184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:09.124956 13184 hybrid_clock.cc:648] HybridClock initialized: now 1781836029124934 us; error 40 us; skew 500 ppm
Jun 19 02:27:09 dist-test-slave-xkrt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:09.128597 13184 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:27:09.129604 13184 webserver.cc:460] Webserver started at http://127.0.98.1:43487/ using document root <none> and password file <none>
I20260619 02:27:09.130327 13184 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:09.130427 13184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:27:09 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:27:09.161615 13184 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:27:09.165727 13230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:09.167310 13184 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 02:27:09.167484 13184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "5c7574c570344c3197342f24bed4120b"
format_stamp: "Formatted at 2026-06-19 02:27:07 on dist-test-slave-xkrt"
server_key: "c994b0cf55d0b4d57213fce2d31b2a6d"
server_key_iv: "f0ee9f50a451c6419728727192009fd9"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:09.168292 13184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:09.191862 13184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:09.195601 13184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:09.195878 13184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:09.196604 13184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:09.197724 13184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:09.197841 13184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:09.197974 13184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:09.198069 13184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:09.210840 13184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41919
I20260619 02:27:09.212395 13184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:27:09.214064 13343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41919 every 8 connection(s)
I20260619 02:27:09.220342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 13184
I20260619 02:27:09.220813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59721/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35725
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39375
--webserver_port=41347
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59721/kms with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:27:09 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:09.247417 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:09.216557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39375 (local address 127.0.98.1:53437)
0619 02:27:09.217992 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:09.218047 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:09.219086 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:09.222228 (+ 3142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:09.222237 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:09.222559 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:09.223197 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:09.223215 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:09.225166 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:09.225170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:09.226614 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:09.226626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:09.227690 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:09.227696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:09.227946 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:09.228820 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:09.228847 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:09.230619 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:09.233301 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:09.233308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:09.233325 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:09.233708 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:09.240648 (+ 6940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:09.240652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:09.240654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:09.240851 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:09.245239 (+ 4388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:09.245247 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:09.245663 (+ 416us) client_negotiation.cc:769] Sending connection context
0619 02:27:09.245871 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 02:27:09.246244 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"thread_start_us":133,"threads_started":1}
I20260619 02:27:09.249588 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:09.217309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53437 (local address 127.0.98.62:39375)
0619 02:27:09.217817 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:09.217840 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:09.218185 (+ 345us) server_negotiation.cc:407] Connection header received
0619 02:27:09.219567 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:09.219581 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:09.219979 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:09.221730 (+ 1751us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:09.223400 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:09.224997 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:09.226770 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:09.227500 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:09.227675 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:09.230865 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:09.230872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:09.230888 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:09.230922 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:09.233165 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:09.240365 (+ 7200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:09.240372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:09.240379 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:09.240544 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:09.241181 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:09.241185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:09.241187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:09.241729 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:09.248195 (+ 6466us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:09.248376 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 02:27:09.248668 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":128,"threads_started":1}
I20260619 02:27:09.252277 13344 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39375
I20260619 02:27:09.252543 13344 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:09.253090 13344 heartbeater.cc:511] Master 127.0.98.62:39375 requested a full tablet report, sending...
W20260619 02:27:09.255400 13344 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39375 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:27:09.255517 13344 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:09.255681 13344 heartbeater.cc:511] Master 127.0.98.62:39375 requested a full tablet report, sending...
I20260619 02:27:09.256538 13344 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:09.256693 13344 heartbeater.cc:511] Master 127.0.98.62:39375 requested a full tablet report, sending...
W20260619 02:27:09.257395 13344 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:27:09.368170 13348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:09.368446 13348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:09.368513 13348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:09.372591 13348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:09.372663 13348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:09.372763 13348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:27:09.379647 13348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59721/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41347
--tserver_master_addrs=127.0.98.62:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:09.381186 13348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:09.382380 13348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:09.393173 13354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:09.393843 13356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:09.393945 13348 server_base.cc:1034] running on GCE node
W20260619 02:27:09.397712 13353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:09.398228 13348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:09.398986 13348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:09.402240 13348 hybrid_clock.cc:648] HybridClock initialized: now 1781836029402203 us; error 46 us; skew 500 ppm
Jun 19 02:27:09 dist-test-slave-xkrt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:09.405515 13348 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:27:09.406677 13348 webserver.cc:460] Webserver started at http://127.0.98.2:41347/ using document root <none> and password file <none>
I20260619 02:27:09.407236 13348 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:09.407337 13348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:27:09 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:27:09.431496 13348 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:27:09.434561 13364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:09.436147 13348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:27:09.436307 13348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "9c87dd22e022460fb17b8862fe49ddf9"
format_stamp: "Formatted at 2026-06-19 02:27:08 on dist-test-slave-xkrt"
server_key: "a51d6c798d0cbe6d939a7ba6e10c4441"
server_key_iv: "c458b43e3a9a8fc4d36531eaf2ebac1d"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:09.437098 13348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:09.467154 13348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:09.470541 13348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:09.470752 13348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:09.471465 13348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:09.472517 13348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:09.472616 13348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:09.472709 13348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:09.472775 13348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:09.483680 13348 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35725
I20260619 02:27:09.484967 13348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:27:09.486186 13478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35725 every 8 connection(s)
I20260619 02:27:09.494100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 13348
I20260619 02:27:09.494566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59721/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44825
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39375
--webserver_port=40861
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59721/kms with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:27:09 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:09.501029 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:09.487376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39375 (local address 127.0.98.2:53405)
0619 02:27:09.488375 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:09.488415 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:09.489239 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:09.489779 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:09.489785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:09.490140 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:09.490607 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:09.490620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:09.491413 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:09.491418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:09.492319 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:09.492328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:09.492842 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:09.492846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:09.493002 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:09.493663 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:09.493679 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:09.495406 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:09.497734 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:09.497739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:09.497752 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:09.498089 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:09.498431 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:09.498435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:09.498437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:09.498633 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:09.499161 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:09.499167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:09.499492 (+ 325us) client_negotiation.cc:769] Sending connection context
0619 02:27:09.499688 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 02:27:09.500011 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":129,"threads_started":1}
I20260619 02:27:09.502279 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:09.486939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53405 (local address 127.0.98.62:39375)
0619 02:27:09.487094 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:09.487098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:09.488552 (+ 1454us) server_negotiation.cc:407] Connection header received
0619 02:27:09.489447 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:09.489451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:09.489508 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:09.489618 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:09.490752 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:09.491281 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:09.492447 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:09.492689 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:09.492801 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:09.495600 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:09.495607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:09.495613 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:09.495646 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:09.497631 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:09.498211 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:09.498215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:09.498216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:09.498303 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:09.498759 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:09.498762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:09.498764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:09.498994 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:09.499106 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:09.501995 (+ 2889us) server_negotiation.cc:299] Negotiation successful
0619 02:27:09.502123 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:27:09.503463 13479 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39375
I20260619 02:27:09.503753 13479 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:09.504433 13479 heartbeater.cc:511] Master 127.0.98.62:39375 requested a full tablet report, sending...
W20260619 02:27:09.506196 13479 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39375 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:27:09.506314 13479 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:09.506458 13479 heartbeater.cc:511] Master 127.0.98.62:39375 requested a full tablet report, sending...
I20260619 02:27:09.507341 13479 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:09.507546 13479 heartbeater.cc:511] Master 127.0.98.62:39375 requested a full tablet report, sending...
W20260619 02:27:09.508381 13479 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:27:09.626967 13482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:09.627410 13482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:09.627537 13482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:09.633625 13482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:09.633752 13482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:09.633880 13482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:27:09.641640 13482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59721/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40861
--tserver_master_addrs=127.0.98.62:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:09.642951 13482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:09.644294 13482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:09.659533 13488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:09.659804 13490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:09.660729 13482 server_base.cc:1034] running on GCE node
W20260619 02:27:09.661706 13487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:09.662254 13482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:09.663048 13482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:09.664233 13482 hybrid_clock.cc:648] HybridClock initialized: now 1781836029664199 us; error 47 us; skew 500 ppm
Jun 19 02:27:09 dist-test-slave-xkrt krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:09.666996 13482 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:27:09.667956 13482 webserver.cc:460] Webserver started at http://127.0.98.3:40861/ using document root <none> and password file <none>
I20260619 02:27:09.668555 13482 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:09.668654 13482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:27:09 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:27:09.692312 13482 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:27:09.696002 13498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:09.697692 13482 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 02:27:09.697897 13482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "c51407af537a484b8c1a8a4cfa1519df"
format_stamp: "Formatted at 2026-06-19 02:27:08 on dist-test-slave-xkrt"
server_key: "82fcd1bfde914c1b8b33f22f58ffe5e3"
server_key_iv: "59fc999edc10aa4d8bc940999e0b6898"
server_key_version: "kuduclusterkey@0"
I20260619 02:27:09.698696 13482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:09.725507 13482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:09.728659 13482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:09.728979 13482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:09.729847 13482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:09.730849 13482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:09.730927 13482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:09.731001 13482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:09.731041 13482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:09.742637 13482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44825
I20260619 02:27:09.744256 13482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:27:09.747659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 13482
I20260619 02:27:09.763232 13611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44825 every 8 connection(s)
I20260619 02:27:09.767656 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:09.748399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.0.98.62:39375)
0619 02:27:09.748551 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:09.748556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:09.749887 (+ 1331us) server_negotiation.cc:407] Connection header received
0619 02:27:09.751017 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:09.751021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:09.751083 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:09.751148 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:09.757207 (+ 6059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:09.758128 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:09.761288 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:09.761759 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:09.761855 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:09.763901 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:09.763905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:09.763908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:09.763935 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:09.765725 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:09.766196 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:09.766200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:09.766202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:09.766298 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:09.766606 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:09.766611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:09.766612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:09.766829 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:09.766991 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:09.767359 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 02:27:09.767490 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 02:27:09 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:09.781445 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:09.767864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38385 (local address 127.0.98.62:39375)
0619 02:27:09.768000 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:09.768004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:09.769226 (+ 1222us) server_negotiation.cc:407] Connection header received
0619 02:27:09.770543 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:09.770546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:09.770591 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:09.770661 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:09.771688 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:09.772186 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:09.773379 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:09.773706 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:09.773801 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:09.777231 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:09.777235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:09.777237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:09.777262 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:09.778950 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:09.779559 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:09.779563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:09.779565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:09.779655 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:09.780082 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:09.780085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:09.780087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:09.780306 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:09.780381 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:09.781162 (+ 781us) server_negotiation.cc:299] Negotiation successful
0619 02:27:09.781280 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:27:09.783176 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:09.766351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39375 (local address 127.0.98.3:38385)
0619 02:27:09.769071 (+ 2720us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:09.769129 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:09.770344 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:09.770834 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:09.770840 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:09.771087 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:09.771557 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:09.771570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:09.772308 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:09.772312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:09.773265 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:09.773274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:09.773863 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:09.773869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:09.774070 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:09.774728 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:09.774746 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:09.777038 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:09.779072 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:09.779078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:09.779093 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:09.779424 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:09.779773 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:09.779776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:09.779778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:09.779961 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:09.780487 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:09.780493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:09.780871 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 02:27:09.781647 (+ 776us) client_negotiation.cc:240] Negotiation successful
0619 02:27:09.781983 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2351,"thread_start_us":181,"threads_started":1}
I20260619 02:27:09.784708 13612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39375
I20260619 02:27:09.784956 13612 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:09.785498 13612 heartbeater.cc:511] Master 127.0.98.62:39375 requested a full tablet report, sending...
I20260619 02:27:09.786993 13146 ts_manager.cc:194] Registered new tserver with Master: c51407af537a484b8c1a8a4cfa1519df (127.0.98.3:44825)
I20260619 02:27:09.788650 13146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38385
Jun 19 02:27:10 dist-test-slave-xkrt krb5kdc[12148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781836030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:10.257645 13344 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:10.257967 13344 heartbeater.cc:511] Master 127.0.98.62:39375 requested a full tablet report, sending...
I20260619 02:27:10.258797 13146 ts_manager.cc:194] Registered new tserver with Master: 5c7574c570344c3197342f24bed4120b (127.0.98.1:41919)
I20260619 02:27:10.259558 13146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53437
Jun 19 02:27:10 dist-test-slave-xkrt krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 02:27:10.508585 13479 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:10.508883 13479 heartbeater.cc:511] Master 127.0.98.62:39375 requested a full tablet report, sending...
I20260619 02:27:10.509666 13146 ts_manager.cc:194] Registered new tserver with Master: 9c87dd22e022460fb17b8862fe49ddf9 (127.0.98.2:35725)
I20260619 02:27:10.510476 13146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53405
I20260619 02:27:10.512552 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:27:10.536747 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:10.514386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40440 (local address 127.0.98.62:39375)
0619 02:27:10.521913 (+ 7527us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:10.521918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:10.521935 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:27:10.521983 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:10.521986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:10.522037 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:10.523352 (+ 1315us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:10.523465 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.524954 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.526467 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.526818 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.527346 (+ 528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:10.528611 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:10.528614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:10.528618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:10.528643 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:10.531915 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:10.534817 (+ 2902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:10.534821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:10.534823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:10.534960 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:10.535290 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:10.535293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:10.535295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:10.535526 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:10.536349 (+ 823us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:10.536430 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 02:27:10.536558 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7429,"thread_start_us":105,"threads_started":1}
I20260619 02:27:10.541930 13146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:27:10.546730 13146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:27:10.585405 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:10.569174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35725 (local address 127.0.0.1:56816)
0619 02:27:10.572156 (+ 2982us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:10.572203 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:10.572322 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:10.574272 (+ 1950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:10.574277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:10.574302 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:10.574435 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:10.574441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.578717 (+ 4276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.578720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:10.580707 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:10.580718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.584877 (+ 4159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.584881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:10.584994 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:10.585040 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 02:27:10.585132 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:27:10.585253 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2816,"thread_start_us":137,"threads_started":1}
I20260619 02:27:10.586294 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:10.569285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.0.98.2:35725)
0619 02:27:10.573052 (+ 3767us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:10.573059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:10.573082 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:27:10.573156 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:10.573162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:10.574110 (+ 948us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:10.574577 (+ 467us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:10.574715 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.575813 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.580866 (+ 5053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.581684 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.585830 (+ 4146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:10.585992 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:10.586044 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 02:27:10.586155 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3664,"thread_start_us":96,"threads_started":1}
I20260619 02:27:10.587467 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:10.570829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41919 (local address 127.0.0.1:48770)
0619 02:27:10.572521 (+ 1692us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:10.572532 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:10.572592 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:10.577239 (+ 4647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:10.577242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:10.577259 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:10.577345 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:10.577350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.581820 (+ 4470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.581824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:10.583762 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:10.583772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.587118 (+ 3346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.587122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:10.587217 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:10.587239 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:27:10.587281 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 02:27:10.587345 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1639,"thread_start_us":82,"threads_started":1}
I20260619 02:27:10.588783 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:10.571208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.0.98.1:41919)
0619 02:27:10.576049 (+ 4841us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:10.576055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:10.576694 (+ 639us) server_negotiation.cc:407] Connection header received
0619 02:27:10.576786 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:10.576791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:10.576979 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:10.577103 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:10.577457 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.578584 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.583952 (+ 5368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.584721 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.588376 (+ 3655us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:10.588522 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:10.588569 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 02:27:10.588657 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4742,"thread_start_us":62,"threads_started":1}
I20260619 02:27:10.589589 13407 tablet_service.cc:1467] Processing CreateTablet for tablet 752a27b9de7a4892b287bbdabe4ca1f3 (DEFAULT_TABLE table=test-table [id=36f7bf3d6ffd4608ad5eeca4417b3bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:10.590734 13407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752a27b9de7a4892b287bbdabe4ca1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:10.594125 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:10.571271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44825 (local address 127.0.0.1:54210)
0619 02:27:10.572769 (+ 1498us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:10.572780 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:10.572837 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:10.588990 (+ 16153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:10.588994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:10.589013 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:10.589094 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:10.589099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.590625 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.590629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:10.591753 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:10.591764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.592854 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.592858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:10.592971 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:10.593007 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:27:10.593068 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 02:27:10.593982 (+ 914us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1441,"thread_start_us":84,"threads_started":1}
I20260619 02:27:10.594805 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:10.571441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54210 (local address 127.0.98.3:44825)
0619 02:27:10.586616 (+ 15175us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:10.586624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:10.586651 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:27:10.586728 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:10.586734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:10.586956 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:10.589246 (+ 2290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:10.589387 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.590492 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.591920 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.592701 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.594385 (+ 1684us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:10.594522 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:10.594572 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 02:27:10.594657 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15061,"thread_start_us":105,"threads_started":1}
I20260619 02:27:10.596408 13272 tablet_service.cc:1467] Processing CreateTablet for tablet 752a27b9de7a4892b287bbdabe4ca1f3 (DEFAULT_TABLE table=test-table [id=36f7bf3d6ffd4608ad5eeca4417b3bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:10.597450 13272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752a27b9de7a4892b287bbdabe4ca1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:10.597504 13546 tablet_service.cc:1467] Processing CreateTablet for tablet 752a27b9de7a4892b287bbdabe4ca1f3 (DEFAULT_TABLE table=test-table [id=36f7bf3d6ffd4608ad5eeca4417b3bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:10.598491 13546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752a27b9de7a4892b287bbdabe4ca1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:10.606438 13644 tablet_bootstrap.cc:492] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9: Bootstrap starting.
I20260619 02:27:10.609056 13644 tablet_bootstrap.cc:654] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:10.609853 13645 tablet_bootstrap.cc:492] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b: Bootstrap starting.
I20260619 02:27:10.612154 13644 log.cc:824] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:10.613014 13646 tablet_bootstrap.cc:492] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df: Bootstrap starting.
I20260619 02:27:10.615031 13646 tablet_bootstrap.cc:654] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:10.615864 13646 log.cc:824] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:10.616606 13645 tablet_bootstrap.cc:654] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:10.617358 13645 log.cc:824] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:10.619467 13644 tablet_bootstrap.cc:492] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9: No bootstrap required, opened a new log
I20260619 02:27:10.619647 13644 ts_tablet_manager.cc:1397] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260619 02:27:10.623713 13644 raft_consensus.cc:348] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c87dd22e022460fb17b8862fe49ddf9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "c51407af537a484b8c1a8a4cfa1519df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44825 } } peers { permanent_uuid: "5c7574c570344c3197342f24bed4120b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41919 } }
I20260619 02:27:10.623929 13644 raft_consensus.cc:374] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:10.623960 13644 raft_consensus.cc:729] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c87dd22e022460fb17b8862fe49ddf9, State: Initialized, Role: FOLLOWER
I20260619 02:27:10.624365 13644 consensus_queue.cc:260] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c87dd22e022460fb17b8862fe49ddf9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "c51407af537a484b8c1a8a4cfa1519df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44825 } } peers { permanent_uuid: "5c7574c570344c3197342f24bed4120b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41919 } }
I20260619 02:27:10.625155 13644 ts_tablet_manager.cc:1428] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:27:10.625494 13479 heartbeater.cc:503] Master 127.0.98.62:39375 was elected leader, sending a full tablet report...
I20260619 02:27:10.627518 13646 tablet_bootstrap.cc:492] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df: No bootstrap required, opened a new log
I20260619 02:27:10.627732 13646 ts_tablet_manager.cc:1397] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260619 02:27:10.631517 13645 tablet_bootstrap.cc:492] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b: No bootstrap required, opened a new log
I20260619 02:27:10.631698 13645 ts_tablet_manager.cc:1397] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20260619 02:27:10.632053 13646 raft_consensus.cc:348] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c87dd22e022460fb17b8862fe49ddf9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "c51407af537a484b8c1a8a4cfa1519df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44825 } } peers { permanent_uuid: "5c7574c570344c3197342f24bed4120b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41919 } }
I20260619 02:27:10.632304 13646 raft_consensus.cc:374] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:10.632354 13646 raft_consensus.cc:729] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c51407af537a484b8c1a8a4cfa1519df, State: Initialized, Role: FOLLOWER
I20260619 02:27:10.632978 13646 consensus_queue.cc:260] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c87dd22e022460fb17b8862fe49ddf9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "c51407af537a484b8c1a8a4cfa1519df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44825 } } peers { permanent_uuid: "5c7574c570344c3197342f24bed4120b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41919 } }
I20260619 02:27:10.634093 13646 ts_tablet_manager.cc:1428] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:27:10.634533 13612 heartbeater.cc:503] Master 127.0.98.62:39375 was elected leader, sending a full tablet report...
I20260619 02:27:10.635619 13645 raft_consensus.cc:348] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c87dd22e022460fb17b8862fe49ddf9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "c51407af537a484b8c1a8a4cfa1519df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44825 } } peers { permanent_uuid: "5c7574c570344c3197342f24bed4120b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41919 } }
I20260619 02:27:10.635865 13645 raft_consensus.cc:374] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:10.635921 13645 raft_consensus.cc:729] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7574c570344c3197342f24bed4120b, State: Initialized, Role: FOLLOWER
I20260619 02:27:10.636359 13645 consensus_queue.cc:260] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c87dd22e022460fb17b8862fe49ddf9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "c51407af537a484b8c1a8a4cfa1519df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44825 } } peers { permanent_uuid: "5c7574c570344c3197342f24bed4120b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41919 } }
I20260619 02:27:10.636982 13344 heartbeater.cc:503] Master 127.0.98.62:39375 was elected leader, sending a full tablet report...
I20260619 02:27:10.637331 13645 ts_tablet_manager.cc:1428] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 02:27:10.697019 13651 raft_consensus.cc:482] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:27:10.697254 13651 raft_consensus.cc:504] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c87dd22e022460fb17b8862fe49ddf9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "c51407af537a484b8c1a8a4cfa1519df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44825 } } peers { permanent_uuid: "5c7574c570344c3197342f24bed4120b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41919 } }
I20260619 02:27:10.698518 13651 leader_election.cc:288] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c87dd22e022460fb17b8862fe49ddf9 (127.0.98.2:35725), 5c7574c570344c3197342f24bed4120b (127.0.98.1:41919)
I20260619 02:27:10.704499 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:10.699155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46841 (local address 127.0.98.1:41919)
0619 02:27:10.699269 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:10.699273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:10.699622 (+ 349us) server_negotiation.cc:407] Connection header received
0619 02:27:10.699795 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:10.699798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:10.699863 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:10.699909 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:10.700382 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.701165 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.702993 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.703609 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.703692 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:10.703753 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:10.704271 (+ 518us) server_negotiation.cc:299] Negotiation successful
0619 02:27:10.704366 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:27:10.704499 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:10.699080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53055 (local address 127.0.98.2:35725)
0619 02:27:10.699209 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:10.699214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:10.699555 (+ 341us) server_negotiation.cc:407] Connection header received
0619 02:27:10.699774 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:10.699777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:10.699829 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:10.699878 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:10.700390 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.701165 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.702993 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.703603 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.703692 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:10.703753 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:10.704316 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 02:27:10.704392 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:27:10.705003 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:10.698795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41919 (local address 127.0.98.3:46841)
0619 02:27:10.699542 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:10.699555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:10.699677 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:10.700026 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:10.700029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:10.700114 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:10.700249 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:10.700256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.701308 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.701311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:10.702870 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:10.702875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.703863 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.703865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:10.704111 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:10.704157 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 02:27:10.704688 (+ 531us) client_negotiation.cc:240] Negotiation successful
0619 02:27:10.704776 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":470,"spinlock_wait_cycles":19200,"thread_start_us":97,"threads_started":1}
I20260619 02:27:10.705416 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:10.698679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35725 (local address 127.0.98.3:53055)
0619 02:27:10.699464 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:10.699479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:10.699642 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:10.700026 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:10.700028 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:10.700059 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:10.700254 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:10.700260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.701308 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.701311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:10.702851 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:10.702862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.703854 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.703857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:10.704056 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:10.704147 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 02:27:10.704970 (+ 823us) client_negotiation.cc:240] Negotiation successful
0619 02:27:10.705255 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":171,"thread_start_us":115,"threads_started":1}
I20260619 02:27:10.705725 13287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752a27b9de7a4892b287bbdabe4ca1f3" candidate_uuid: "c51407af537a484b8c1a8a4cfa1519df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7574c570344c3197342f24bed4120b" is_pre_election: true
I20260619 02:27:10.706058 13287 raft_consensus.cc:2454] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c51407af537a484b8c1a8a4cfa1519df in term 0.
I20260619 02:27:10.706099 13420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752a27b9de7a4892b287bbdabe4ca1f3" candidate_uuid: "c51407af537a484b8c1a8a4cfa1519df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c87dd22e022460fb17b8862fe49ddf9" is_pre_election: true
I20260619 02:27:10.706350 13420 raft_consensus.cc:2454] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c51407af537a484b8c1a8a4cfa1519df in term 0.
I20260619 02:27:10.706533 13500 leader_election.cc:302] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c7574c570344c3197342f24bed4120b, c51407af537a484b8c1a8a4cfa1519df; no voters:
I20260619 02:27:10.706827 13651 raft_consensus.cc:2781] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:27:10.706918 13651 raft_consensus.cc:482] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:27:10.706961 13651 raft_consensus.cc:3037] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:10.708000 13651 raft_consensus.cc:504] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c87dd22e022460fb17b8862fe49ddf9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "c51407af537a484b8c1a8a4cfa1519df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44825 } } peers { permanent_uuid: "5c7574c570344c3197342f24bed4120b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41919 } }
I20260619 02:27:10.708750 13420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752a27b9de7a4892b287bbdabe4ca1f3" candidate_uuid: "c51407af537a484b8c1a8a4cfa1519df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c87dd22e022460fb17b8862fe49ddf9"
I20260619 02:27:10.708750 13287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752a27b9de7a4892b287bbdabe4ca1f3" candidate_uuid: "c51407af537a484b8c1a8a4cfa1519df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7574c570344c3197342f24bed4120b"
I20260619 02:27:10.708870 13420 raft_consensus.cc:3037] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:10.708904 13287 raft_consensus.cc:3037] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:10.708385 13651 leader_election.cc:288] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [CANDIDATE]: Term 1 election: Requested vote from peers 9c87dd22e022460fb17b8862fe49ddf9 (127.0.98.2:35725), 5c7574c570344c3197342f24bed4120b (127.0.98.1:41919)
I20260619 02:27:10.710157 13420 raft_consensus.cc:2454] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c51407af537a484b8c1a8a4cfa1519df in term 1.
I20260619 02:27:10.710271 13287 raft_consensus.cc:2454] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c51407af537a484b8c1a8a4cfa1519df in term 1.
I20260619 02:27:10.710691 13501 leader_election.cc:302] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5c7574c570344c3197342f24bed4120b, 9c87dd22e022460fb17b8862fe49ddf9, c51407af537a484b8c1a8a4cfa1519df; no voters:
I20260619 02:27:10.710911 13651 raft_consensus.cc:2781] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:27:10.711185 13651 raft_consensus.cc:686] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [term 1 LEADER]: Becoming Leader. State: Replica: c51407af537a484b8c1a8a4cfa1519df, State: Running, Role: LEADER
I20260619 02:27:10.711562 13651 consensus_queue.cc:237] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c87dd22e022460fb17b8862fe49ddf9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "c51407af537a484b8c1a8a4cfa1519df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44825 } } peers { permanent_uuid: "5c7574c570344c3197342f24bed4120b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41919 } }
I20260619 02:27:10.715165 13146 catalog_manager.cc:5495] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df reported cstate change: term changed from 0 to 1, leader changed from <none> to c51407af537a484b8c1a8a4cfa1519df (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c51407af537a484b8c1a8a4cfa1519df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c87dd22e022460fb17b8862fe49ddf9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c51407af537a484b8c1a8a4cfa1519df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c7574c570344c3197342f24bed4120b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41919 } health_report { overall_health: UNKNOWN } } }
W20260619 02:27:10.716274 13345 tablet.cc:2369] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:27:10.728894 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:10.724997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.0.98.3:44825)
0619 02:27:10.725123 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:10.725128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:10.725205 (+ 77us) server_negotiation.cc:407] Connection header received
0619 02:27:10.725408 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:10.725412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:10.725476 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:10.725619 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:27:10.725883 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.726430 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.727364 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:10.727612 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:10.727735 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:10.728041 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:27:10.728454 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:27:10.728589 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:10.728672 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 02:27:10.728737 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:27:10.735679 13287 raft_consensus.cc:1261] T 752a27b9de7a4892b287bbdabe4ca1f3 P 5c7574c570344c3197342f24bed4120b [term 1 FOLLOWER]: Refusing update from remote peer c51407af537a484b8c1a8a4cfa1519df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:27:10.735877 13420 raft_consensus.cc:1261] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9 [term 1 FOLLOWER]: Refusing update from remote peer c51407af537a484b8c1a8a4cfa1519df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:27:10.736502 13651 consensus_queue.cc:1035] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7574c570344c3197342f24bed4120b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:27:10.736764 13655 consensus_queue.cc:1035] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c87dd22e022460fb17b8862fe49ddf9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:27:10.740743 13480 tablet.cc:2369] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:27:10.744619 13658 mvcc.cc:205] Tried to move back new op lower bound from 7298400381885853696 to 7298400381797974016. Current Snapshot: MvccSnapshot[applied={T|T < 7298400381885853696}]
I20260619 02:27:10.747717 13659 mvcc.cc:205] Tried to move back new op lower bound from 7298400381885853696 to 7298400381797974016. Current Snapshot: MvccSnapshot[applied={T|T < 7298400381885853696}]
I20260619 02:27:10.755136 13660 mvcc.cc:205] Tried to move back new op lower bound from 7298400381885853696 to 7298400381797974016. Current Snapshot: MvccSnapshot[applied={T|T < 7298400381885853696}]
I20260619 02:27:10.759106 13145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:27:10.759311 13145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:27:10.762120 13145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab46bec829004c52b815af7df87222f6: Sending DeleteTablet for 3 replicas of tablet 752a27b9de7a4892b287bbdabe4ca1f3
I20260619 02:27:10.763569 13546 tablet_service.cc:1514] Processing DeleteTablet for tablet 752a27b9de7a4892b287bbdabe4ca1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54210
I20260619 02:27:10.763903 13407 tablet_service.cc:1514] Processing DeleteTablet for tablet 752a27b9de7a4892b287bbdabe4ca1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56816
I20260619 02:27:10.764611 13272 tablet_service.cc:1514] Processing DeleteTablet for tablet 752a27b9de7a4892b287bbdabe4ca1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48770
I20260619 02:27:10.766274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 13184
I20260619 02:27:10.769723 13668 tablet_replica.cc:330] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9: stopping tablet replica
I20260619 02:27:10.770041 13668 raft_consensus.cc:2229] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:27:10.770349 13668 raft_consensus.cc:2258] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:27:10.779125 13668 ts_tablet_manager.cc:1905] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:27:10.782605 13668 ts_tablet_manager.cc:1918] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:27:10.782716 13668 log.cc:1196] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/752a27b9de7a4892b287bbdabe4ca1f3
I20260619 02:27:10.783046 13668 ts_tablet_manager.cc:1939] T 752a27b9de7a4892b287bbdabe4ca1f3 P 9c87dd22e022460fb17b8862fe49ddf9: Deleting consensus metadata
I20260619 02:27:10.783069 13667 tablet_replica.cc:330] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df: stopping tablet replica
W20260619 02:27:10.783111 13131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41919 (error 108)
I20260619 02:27:10.783347 13667 raft_consensus.cc:2229] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:27:10.783700 13667 raft_consensus.cc:2258] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:27:10.784332 13132 catalog_manager.cc:4841] TS 9c87dd22e022460fb17b8862fe49ddf9 (127.0.98.2:35725): tablet 752a27b9de7a4892b287bbdabe4ca1f3 (table test-table [id=36f7bf3d6ffd4608ad5eeca4417b3bb8]) successfully deleted
I20260619 02:27:10.785095 13667 ts_tablet_manager.cc:1905] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:27:10.787650 13667 ts_tablet_manager.cc:1918] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:27:10.787742 13667 log.cc:1196] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/752a27b9de7a4892b287bbdabe4ca1f3
I20260619 02:27:10.788038 13667 ts_tablet_manager.cc:1939] T 752a27b9de7a4892b287bbdabe4ca1f3 P c51407af537a484b8c1a8a4cfa1519df: Deleting consensus metadata
I20260619 02:27:10.789139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 13348
I20260619 02:27:10.789116 13132 catalog_manager.cc:4841] TS c51407af537a484b8c1a8a4cfa1519df (127.0.98.3:44825): tablet 752a27b9de7a4892b287bbdabe4ca1f3 (table test-table [id=36f7bf3d6ffd4608ad5eeca4417b3bb8]) successfully deleted
I20260619 02:27:10.794476 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:10.794155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41919 (local address 127.0.0.1:48780)
0619 02:27:10.794311 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:10.794388 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 02:27:10.794656 13131 catalog_manager.cc:4584] TS 5c7574c570344c3197342f24bed4120b (127.0.98.1:41919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 752a27b9de7a4892b287bbdabe4ca1f3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41919: connect: Connection refused (error 111)
I20260619 02:27:10.798324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 13482
I20260619 02:27:10.806628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 13112
I20260619 02:27:10.815690 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 02:27:11.149780 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 02:27:11.149909 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 02:27:11.494104 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 02:27:11.494 UTC [12178] LOG: received smart shutdown request
2026-06-19 02:27:11.497 UTC [12178] LOG: background worker "logical replication launcher" (PID 12187) exited with exit code 1
2026-06-19 02:27:11.498 UTC [12182] LOG: shutting down
2026-06-19 02:27:11.655 UTC [12178] LOG: database system is shut down
2026-06-19T02:27:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48335 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:27:11 dist-test-slave-xkrt krb5kdc[13679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:27:11 dist-test-slave-xkrt krb5kdc[13679](info): set up 2 sockets
Jun 19 02:27:11 dist-test-slave-xkrt krb5kdc[13679](info): commencing operation
krb5kdc: starting...
W20260619 02:27:12.813644 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:27:12 dist-test-slave-xkrt krb5kdc[13679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:27:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:27:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:27:12.932539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39139
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39139,127.0.98.61:46739,127.0.98.60:45739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:13.041189 13692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:13.041441 13692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:13.041487 13692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:13.045360 13692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:27:13.045415 13692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:13.045434 13692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:13.045451 13692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:27:13.045466 13692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:27:13.050640 13692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39139,127.0.98.61:46739,127.0.98.60:45739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:13.051646 13692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:13.052610 13692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:13.060981 13700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:13.060936 13698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:13.060936 13697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:13.061201 13692 server_base.cc:1034] running on GCE node
I20260619 02:27:13.061789 13692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:13.062904 13692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:13.064095 13692 hybrid_clock.cc:648] HybridClock initialized: now 1781836033064083 us; error 27 us; skew 500 ppm
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:13.066781 13692 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:27:13.067618 13692 webserver.cc:460] Webserver started at http://127.0.98.62:46417/ using document root <none> and password file <none>
I20260619 02:27:13.068154 13692 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:13.068238 13692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:13.068470 13692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:27:13.070511 13692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "8246b7e7ee264e38b9b7e2662cf20bd0"
format_stamp: "Formatted at 2026-06-19 02:27:13 on dist-test-slave-xkrt"
server_key: "fc0aabd44558f42726067708529e0b36"
server_key_iv: "db116111732c8a9c7d7db3b13b3385a5"
server_key_version: "encryptionkey@0"
I20260619 02:27:13.071055 13692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "8246b7e7ee264e38b9b7e2662cf20bd0"
format_stamp: "Formatted at 2026-06-19 02:27:13 on dist-test-slave-xkrt"
server_key: "fc0aabd44558f42726067708529e0b36"
server_key_iv: "db116111732c8a9c7d7db3b13b3385a5"
server_key_version: "encryptionkey@0"
I20260619 02:27:13.074570 13692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:27:13.076830 13707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:13.078058 13692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:27:13.078184 13692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "8246b7e7ee264e38b9b7e2662cf20bd0"
format_stamp: "Formatted at 2026-06-19 02:27:13 on dist-test-slave-xkrt"
server_key: "fc0aabd44558f42726067708529e0b36"
server_key_iv: "db116111732c8a9c7d7db3b13b3385a5"
server_key_version: "encryptionkey@0"
I20260619 02:27:13.078286 13692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:13.090592 13692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:13.093361 13692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:13.093573 13692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:13.100129 13692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39139
I20260619 02:27:13.100189 13759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39139 every 8 connection(s)
I20260619 02:27:13.101051 13692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:27:13.103724 13760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:13.107328 13760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:27:13.108016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 13692
I20260619 02:27:13.108145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:27:13.108443 392 external_mini_cluster.cc:1428] Setting key d62081fe6f72de0d0c2c5d2278b4211c
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:13.125706 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.109794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39139 (local address 127.0.0.1:57254)
0619 02:27:13.110348 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.110394 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:13.111181 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:13.111931 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:13.111935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:13.112273 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:13.112536 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.112548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.114335 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.114338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.115056 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.115064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.115960 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.115964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.116099 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.116667 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:13.116682 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:13.120436 (+ 3754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:13.122715 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.122723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.122730 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.122992 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.123432 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.123434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.123436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.123544 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.124306 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:13.124311 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:13.124426 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 02:27:13.124564 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 02:27:13.124906 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":6,"spinlock_wait_cycles":28416,"thread_start_us":153,"threads_started":1}
I20260619 02:27:13.125712 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.109794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.0.98.62:39139)
0619 02:27:13.110228 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.110255 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.110502 (+ 247us) server_negotiation.cc:407] Connection header received
0619 02:27:13.111469 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.111478 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.111704 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.111911 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.112683 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.114222 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.115173 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.115800 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.116052 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.120581 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.120586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.120597 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.120617 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.122606 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.123159 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.123177 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.123182 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.123329 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.123639 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.123644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.123647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.124190 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.124348 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.124673 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.124988 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":163,"server-negotiator.queue_time_us":269,"spinlock_wait_cycles":3840,"thread_start_us":125,"threads_started":1}
I20260619 02:27:13.128039 13760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:27:13.128685 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.128492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46739 (local address 127.0.0.1:56762)
0619 02:27:13.128574 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.128636 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 02:27:13.128878 13708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46739: connect: Connection refused (error 111)
I20260619 02:27:13.129817 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.129506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46739 (local address 127.0.0.1:56772)
0619 02:27:13.129646 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.129689 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 02:27:13.129962 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46739: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46739: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 02:27:13.136116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46739
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39139,127.0.98.61:46739,127.0.98.60:45739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:27:13.179224 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } attempt: 1
I20260619 02:27:13.180208 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.179853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46739 (local address 127.0.0.1:56780)
0619 02:27:13.180042 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.180116 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 02:27:13.180985 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.180763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46739 (local address 127.0.0.1:56784)
0619 02:27:13.180871 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.180923 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 02:27:13.181140 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46739: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46739: connect: Connection refused (error 111)
W20260619 02:27:13.244668 13767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:13.244910 13767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:13.244978 13767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:13.248946 13767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:27:13.248997 13767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:13.249017 13767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:13.249032 13767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:27:13.249084 13767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:27:13.249341 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } attempt: 2
I20260619 02:27:13.250197 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.249913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46739 (local address 127.0.0.1:56786)
0619 02:27:13.250060 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.250123 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 02:27:13.250867 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.250687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46739 (local address 127.0.0.1:56796)
0619 02:27:13.250774 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.250818 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 02:27:13.251019 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46739: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46739: connect: Connection refused (error 111)
I20260619 02:27:13.254619 13767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39139,127.0.98.61:46739,127.0.98.60:45739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:13.255592 13767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:13.256546 13767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:13.264530 13773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:13.264484 13772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:13.264485 13775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:13.264606 13767 server_base.cc:1034] running on GCE node
I20260619 02:27:13.265211 13767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:13.266283 13767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:13.267470 13767 hybrid_clock.cc:648] HybridClock initialized: now 1781836033267455 us; error 25 us; skew 500 ppm
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:13.270081 13767 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 02:27:13.270849 13767 webserver.cc:460] Webserver started at http://127.0.98.61:35341/ using document root <none> and password file <none>
I20260619 02:27:13.271414 13767 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:13.271502 13767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:13.271713 13767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:27:13.273669 13767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/data/instance:
uuid: "25f81d5df4704d6099da624d9601a088"
format_stamp: "Formatted at 2026-06-19 02:27:13 on dist-test-slave-xkrt"
server_key: "d32be2870bbe3186119f362573b350b0"
server_key_iv: "25cc3a08161e4f47e5accb4824627f87"
server_key_version: "encryptionkey@0"
I20260619 02:27:13.274204 13767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/wal/instance:
uuid: "25f81d5df4704d6099da624d9601a088"
format_stamp: "Formatted at 2026-06-19 02:27:13 on dist-test-slave-xkrt"
server_key: "d32be2870bbe3186119f362573b350b0"
server_key_iv: "25cc3a08161e4f47e5accb4824627f87"
server_key_version: "encryptionkey@0"
I20260619 02:27:13.277633 13767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:27:13.279867 13782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:13.281104 13767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:27:13.281226 13767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/wal
uuid: "25f81d5df4704d6099da624d9601a088"
format_stamp: "Formatted at 2026-06-19 02:27:13 on dist-test-slave-xkrt"
server_key: "d32be2870bbe3186119f362573b350b0"
server_key_iv: "25cc3a08161e4f47e5accb4824627f87"
server_key_version: "encryptionkey@0"
I20260619 02:27:13.281339 13767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:13.297282 13767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:13.300010 13767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:13.300199 13767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:13.306700 13767 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46739
I20260619 02:27:13.306726 13834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46739 every 8 connection(s)
I20260619 02:27:13.307749 13767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/data/info.pb
I20260619 02:27:13.310290 13835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:13.311601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 13767
I20260619 02:27:13.311741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-1/wal/instance
I20260619 02:27:13.312057 392 external_mini_cluster.cc:1428] Setting key f901c8ad21941bac3bb51c0f59997a9a
I20260619 02:27:13.314224 13835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:13.327407 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.316153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.0.98.62:39139)
0619 02:27:13.316301 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.316306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.317100 (+ 794us) server_negotiation.cc:407] Connection header received
0619 02:27:13.318029 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.318035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.318092 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.318205 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.319252 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.320037 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.321115 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.321345 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.321423 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.323890 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.323894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.323896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.323919 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.325414 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.325921 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.325926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.325928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.326003 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.326350 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.326353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.326354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.326523 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.326624 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.327102 (+ 478us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.327217 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:27:13.328114 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.316504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39139 (local address 127.0.0.1:57262)
0619 02:27:13.316937 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.316984 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:13.317819 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:13.318336 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:13.318344 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:13.318646 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:13.319111 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.319123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.320152 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.320155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.321005 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.321014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.321495 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.321499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.321678 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.322295 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:13.322321 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:13.323722 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:13.325526 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.325532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.325565 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.325820 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.326102 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.326106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.326108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.326251 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.326624 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:13.326631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:13.326868 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 02:27:13.327038 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 02:27:13.327344 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":127,"threads_started":1}
I20260619 02:27:13.329417 13835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 02:27:13.337457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45739
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39139,127.0.98.61:46739,127.0.98.60:45739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:27:13.339109 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.329906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46739 (local address 127.0.0.1:56812)
0619 02:27:13.330017 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.330033 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:13.330115 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:13.330588 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:13.330590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:13.330714 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:13.330832 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.330839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.332005 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.332007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.332681 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.332689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.333444 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.333447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.333541 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.334098 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:13.334114 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:13.335431 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:13.337592 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.337597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.337600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.337856 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.338127 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.338133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.338138 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.338237 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.338631 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:13.338634 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:13.338711 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 02:27:13.338819 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 02:27:13.338941 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 02:27:13.339246 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.329997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.0.98.61:46739)
0619 02:27:13.330278 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.330285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.330310 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:27:13.330344 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.330347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.330479 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.330580 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.331179 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.331916 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.332892 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.333332 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.333444 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.335598 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.335610 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.335625 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.335669 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.337462 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.337961 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.337966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.337967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.338038 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.338329 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.338330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.338331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.338532 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.338665 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.338940 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.339056 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":134,"threads_started":1}
I20260619 02:27:13.341054 13835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:27:13.341751 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.341490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49684)
0619 02:27:13.341615 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.341687 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 02:27:13.342016 13785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
I20260619 02:27:13.342156 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } attempt: 3
I20260619 02:27:13.342926 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.342709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49686)
0619 02:27:13.342795 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.342867 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 02:27:13.343194 13835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45739: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 02:27:13.351655 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.342538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46739 (local address 127.0.0.1:56816)
0619 02:27:13.342900 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.342918 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:13.343022 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:13.343316 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:13.343321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:13.343523 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:13.343676 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.343699 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.344717 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.344723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.346270 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.346284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.346727 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.346733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.346875 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.347308 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:13.347329 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:13.348645 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:13.350273 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.350276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.350278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.350469 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.350732 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.350735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.350737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.350817 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.351167 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:13.351172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:13.351264 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 02:27:13.351374 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:27:13.351498 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
I20260619 02:27:13.351653 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.342750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.0.98.61:46739)
0619 02:27:13.342834 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.342838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.343049 (+ 211us) server_negotiation.cc:407] Connection header received
0619 02:27:13.343111 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.343113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.343145 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.343280 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.343800 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.344600 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.346398 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.346613 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.346691 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.348771 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.348775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.348776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.348795 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.350143 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.350561 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.350566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.350568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.350640 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.350912 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.350915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.350917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.351071 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.351164 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.351421 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.351518 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:27:13.352344 13760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:27:13.352963 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.352766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49692)
0619 02:27:13.352868 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.352911 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 02:27:13.353626 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.353377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49708)
0619 02:27:13.353500 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.353570 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 02:27:13.353758 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45739: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
I20260619 02:27:13.384898 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } attempt: 1
I20260619 02:27:13.385782 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.385413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49710)
0619 02:27:13.385635 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.385696 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260619 02:27:13.386521 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.386281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49712)
0619 02:27:13.386389 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.386442 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 02:27:13.386650 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45739: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
I20260619 02:27:13.392426 13835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } attempt: 1
I20260619 02:27:13.393061 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.392835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49716)
0619 02:27:13.392952 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.393000 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 02:27:13.393723 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.393513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49724)
0619 02:27:13.393624 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.393671 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 02:27:13.393850 13835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45739: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
W20260619 02:27:13.449649 13841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:13.449913 13841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:13.449977 13841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:13.454015 13841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:27:13.454083 13841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:13.454119 13841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:13.454169 13841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:27:13.454209 13841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:27:13.459445 13841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39139,127.0.98.61:46739,127.0.98.60:45739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:13.460430 13841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:13.461431 13841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:27:13.461818 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } attempt: 2
I20260619 02:27:13.462006 13835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } attempt: 2
I20260619 02:27:13.462646 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.462329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49734)
0619 02:27:13.462489 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.462557 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 02:27:13.462653 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.462378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49748)
0619 02:27:13.462539 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.462587 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 02:27:13.463356 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.463164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49764)
0619 02:27:13.463257 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.463299 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 02:27:13.463532 13835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45739: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
I20260619 02:27:13.463690 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.463400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49778)
0619 02:27:13.463559 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.463614 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260619 02:27:13.463819 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45739: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45739: connect: Connection refused (error 111)
W20260619 02:27:13.469940 13847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:13.469954 13848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:13.470000 13841 server_base.cc:1034] running on GCE node
W20260619 02:27:13.469940 13850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:13.470768 13841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:13.471781 13841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:13.472944 13841 hybrid_clock.cc:648] HybridClock initialized: now 1781836033472927 us; error 27 us; skew 500 ppm
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:13.475525 13841 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 02:27:13.476280 13841 webserver.cc:460] Webserver started at http://127.0.98.60:42227/ using document root <none> and password file <none>
I20260619 02:27:13.476778 13841 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:13.476833 13841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:13.476994 13841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:27:13.478948 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/data/instance:
uuid: "806bbb007c9642348fba2af16f5da24f"
format_stamp: "Formatted at 2026-06-19 02:27:13 on dist-test-slave-xkrt"
server_key: "71c07951a0d026fc4467531b06b51601"
server_key_iv: "02172e3876b0b11ae975400e919dc8ac"
server_key_version: "encryptionkey@0"
I20260619 02:27:13.479426 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/wal/instance:
uuid: "806bbb007c9642348fba2af16f5da24f"
format_stamp: "Formatted at 2026-06-19 02:27:13 on dist-test-slave-xkrt"
server_key: "71c07951a0d026fc4467531b06b51601"
server_key_iv: "02172e3876b0b11ae975400e919dc8ac"
server_key_version: "encryptionkey@0"
I20260619 02:27:13.483074 13841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:27:13.485311 13857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:13.486526 13841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:27:13.486615 13841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/wal
uuid: "806bbb007c9642348fba2af16f5da24f"
format_stamp: "Formatted at 2026-06-19 02:27:13 on dist-test-slave-xkrt"
server_key: "71c07951a0d026fc4467531b06b51601"
server_key_iv: "02172e3876b0b11ae975400e919dc8ac"
server_key_version: "encryptionkey@0"
I20260619 02:27:13.486688 13841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:13.496798 13841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:13.499678 13841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:13.499812 13841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:13.506283 13841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45739
I20260619 02:27:13.506342 13909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45739 every 8 connection(s)
I20260619 02:27:13.507189 13841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/data/info.pb
I20260619 02:27:13.508104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 13841
I20260619 02:27:13.508230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781835873883564-392-0/minicluster-data/master-2/wal/instance
I20260619 02:27:13.508558 392 external_mini_cluster.cc:1428] Setting key 5bea537b8afa0cd66e4d79312c9f3c2b
I20260619 02:27:13.510353 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:13.513867 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:13.521975 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.509835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49780 (local address 127.0.98.60:45739)
0619 02:27:13.510233 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.510246 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.510279 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:27:13.510936 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.510953 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.511250 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.511577 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.512344 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.513303 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.514194 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.514644 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.514807 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.517408 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.517413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.517423 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.517451 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.519313 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.519776 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.519780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.519784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.519896 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.520165 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.520168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.520169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.520507 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.520726 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.520958 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.521245 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"spinlock_wait_cycles":14720,"thread_start_us":135,"threads_started":1}
I20260619 02:27:13.524516 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.515507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39139 (local address 127.0.0.1:57268)
0619 02:27:13.515925 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.515981 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:13.516113 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:13.516402 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:13.516410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:13.516735 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:13.516851 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.516856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.517695 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.517698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.518585 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.518594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.519149 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.519151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.519230 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.519670 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:13.519685 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:13.521106 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:13.522791 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.522799 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.522804 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.523011 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.523285 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.523289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.523291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.523382 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.523887 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:13.523893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:13.524004 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 02:27:13.524180 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 02:27:13.524345 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":145,"threads_started":1}
I20260619 02:27:13.524559 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.515576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57268 (local address 127.0.98.62:39139)
0619 02:27:13.515722 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.515727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.516071 (+ 344us) server_negotiation.cc:407] Connection header received
0619 02:27:13.516201 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.516203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.516245 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.516319 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.516968 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.517576 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.518697 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.519037 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.519158 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.521238 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.521242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.521244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.521270 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.522667 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.523116 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.523120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.523121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.523198 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.523507 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.523512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.523514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.523724 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.523802 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.524270 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.524397 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:27:13.525489 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 02:27:13.532871 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.523971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.0.98.62:39139)
0619 02:27:13.524285 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.524293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.524305 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:27:13.524335 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.524337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.524382 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.524471 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.525155 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.525922 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.526967 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.527213 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.527325 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.529510 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.529513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.529515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.529535 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.531238 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.531613 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.531619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.531623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.531709 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.531971 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.531973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.531974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.532180 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.532355 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.532541 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.532650 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":101,"threads_started":1}
I20260619 02:27:13.533409 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.525924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46739 (local address 127.0.0.1:56824)
0619 02:27:13.526044 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.526057 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:13.526120 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:13.526322 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:13.526325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:13.526482 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:13.526580 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.526586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.527252 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.527257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.527924 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.527932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.528391 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.528394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.528481 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.528890 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:13.528903 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:13.530087 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:13.531696 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.531699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.531700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.531990 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.532258 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.532261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.532262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.532352 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.532786 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:13.532789 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:13.532875 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 02:27:13.533205 (+ 330us) client_negotiation.cc:240] Negotiation successful
0619 02:27:13.533309 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 02:27:13.533578 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.526015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.0.98.61:46739)
0619 02:27:13.526121 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.526126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.526143 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:27:13.526183 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.526185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.526220 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.526282 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.526698 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.527154 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.528044 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.528271 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.528389 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.530223 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.530228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.530234 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.530267 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.531596 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.532078 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.532082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.532083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.532155 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.532450 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.532453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.532468 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.532675 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.532765 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.533276 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.533390 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:27:13.534442 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 02:27:13.542454 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.534901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49786)
0619 02:27:13.534983 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.534996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:13.535063 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:13.535272 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:13.535274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:13.535393 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:13.535457 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.535461 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.536149 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.536152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.536625 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.536631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.537046 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.537049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.537150 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.537583 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:13.537600 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:13.538778 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:13.541084 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.541090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.541093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.541270 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.541524 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.541528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.541530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.541624 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.541966 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:13.541969 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:13.542045 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 02:27:13.542150 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 02:27:13.542283 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":19}
I20260619 02:27:13.542464 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.534966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.0.98.60:45739)
0619 02:27:13.535088 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.535095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.535108 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:27:13.535141 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.535144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.535180 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.535261 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.535558 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.536043 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.536724 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.536951 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.537049 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.538925 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.538929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.538931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.538952 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.540985 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.541366 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.541369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.541370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.541442 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.541714 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.541719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.541721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.541876 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.541975 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.542210 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.542342 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":51}
I20260619 02:27:13.542725 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.534255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56832 (local address 127.0.98.61:46739)
0619 02:27:13.534358 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.534362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.534374 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:27:13.534481 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.534484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.534525 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.534598 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.535380 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.535841 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.536548 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.536767 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.536868 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.539517 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.539520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.539522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.539544 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.541026 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.541401 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.541406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.541408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.541482 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.541754 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.541758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.541760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.541933 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.542152 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.542254 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.542358 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:27:13.545819 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:27:13.545960 13910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f: Bootstrap starting.
I20260619 02:27:13.548472 13910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:13.549394 13910 log.cc:824] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:13.551499 13910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f: No bootstrap required, opened a new log
I20260619 02:27:13.554661 13835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } attempt: 3
I20260619 02:27:13.556367 13910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } }
I20260619 02:27:13.556623 13910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:13.556679 13910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 806bbb007c9642348fba2af16f5da24f, State: Initialized, Role: FOLLOWER
I20260619 02:27:13.557214 13910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } }
I20260619 02:27:13.558012 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.548099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.0.98.61:46739)
0619 02:27:13.548189 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.548193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.548630 (+ 437us) server_negotiation.cc:407] Connection header received
0619 02:27:13.548787 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.548790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.548828 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.548881 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.549756 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.550466 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.551761 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.552084 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.552153 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.554279 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.554282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.554285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.554309 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.556106 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.556590 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.556596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.556600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.556684 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.557014 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.557018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.557020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.557231 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.557360 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.557731 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.557838 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:27:13.558374 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.547769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57290 (local address 127.0.98.62:39139)
0619 02:27:13.547888 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.547892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.548352 (+ 460us) server_negotiation.cc:407] Connection header received
0619 02:27:13.548458 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.548461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.548510 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.548591 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.549479 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.550268 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.551336 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.551674 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.551750 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.554044 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.554047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.554050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.554075 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.556523 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.557023 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.557027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.557028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.557125 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.557428 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.557432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.557433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.557668 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.557774 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.558117 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.558232 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:27:13.558619 13910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:27:13.560036 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.548009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.0.98.60:45739)
0619 02:27:13.548123 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.548127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.548344 (+ 217us) server_negotiation.cc:407] Connection header received
0619 02:27:13.548498 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.548501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.548542 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.548596 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.549693 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.550444 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.551761 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.552089 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.552153 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.554279 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.554282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.554285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.554309 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.556571 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.557166 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.557170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.557171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.557243 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.557685 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.557689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.557691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.557889 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.557994 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.559805 (+ 1811us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.559922 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:27:13.560973 13925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } } }
I20260619 02:27:13.561844 13925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:27:13.562954 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } attempt: 3
W20260619 02:27:13.564164 13936 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:27:13.564234 13936 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:27:13.562445 13841 master_runner.cc:186] Error getting master registration for 127.0.98.62:39139: OK, instance_id { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" instance_seqno: 1781836033098314 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 02:27:13.566223 13841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 02:27:13 dist-test-slave-xkrt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 02:27:13.571090 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.555035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49802)
0619 02:27:13.555262 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.555276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:13.555349 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:13.560953 (+ 5604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:13.560956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:13.561129 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:13.561196 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.561201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.562490 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.562494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.563244 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.563254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.564581 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.564584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.564664 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.565101 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:13.565117 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:13.566443 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:13.568386 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.568390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.568392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.568642 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.570059 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.570062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.570064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.570149 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.570600 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:13.570605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:13.570699 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 02:27:13.570802 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 02:27:13.570928 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260619 02:27:13.571771 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.555236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.0.98.60:45739)
0619 02:27:13.560482 (+ 5246us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.560496 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.560510 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:27:13.560546 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.560549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.560854 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.560941 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.561302 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.562359 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.563644 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.563990 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.564085 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.566566 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.566569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.566573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.566602 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.568253 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.569839 (+ 1586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.569842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.569844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.569948 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.570253 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.570260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.570263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.570478 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.570579 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.571499 (+ 920us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.571603 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":99,"threads_started":1}
I20260619 02:27:13.572192 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.563259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45739 (local address 127.0.0.1:49818)
0619 02:27:13.563468 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:13.563481 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:13.563546 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:13.564514 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:13.564518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:13.564684 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:13.564773 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.564779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.566011 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.566014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.566630 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:13.566639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.567234 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.567238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:13.567321 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.567776 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:13.567789 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:13.568948 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:13.570838 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.570841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.570843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.571027 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.571309 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:13.571312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:13.571314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:13.571392 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:13.571730 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:13.571733 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:13.571831 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 02:27:13.571944 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 02:27:13.572056 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260619 02:27:13.573622 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:13.564218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.0.98.60:45739)
0619 02:27:13.564312 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:13.564319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:13.564332 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:27:13.564362 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:13.564364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:13.564397 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:13.564479 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:13.565161 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.565887 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.566745 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:13.567103 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:13.567232 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:13.569095 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:13.569098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:13.569101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:13.569123 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:13.570728 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.571120 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.571125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.571126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.571208 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:13.571480 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:13.571485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:13.571487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:13.571640 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:13.571736 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:13.572010 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 02:27:13.572106 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:27:13.574674 13835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088: Bootstrap starting.
I20260619 02:27:13.575757 13760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0: Bootstrap starting.
I20260619 02:27:13.577481 13835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:13.577790 13760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:13.578399 13835 log.cc:824] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:13.578513 13760 log.cc:824] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:13.580407 13835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088: No bootstrap required, opened a new log
I20260619 02:27:13.580407 13760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0: No bootstrap required, opened a new log
I20260619 02:27:13.584470 13760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } }
I20260619 02:27:13.584470 13835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } }
I20260619 02:27:13.584683 13835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:13.584682 13760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:13.584733 13835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f81d5df4704d6099da624d9601a088, State: Initialized, Role: FOLLOWER
I20260619 02:27:13.584733 13760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8246b7e7ee264e38b9b7e2662cf20bd0, State: Initialized, Role: FOLLOWER
I20260619 02:27:13.585184 13835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } }
I20260619 02:27:13.585184 13760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } }
I20260619 02:27:13.586532 13760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:27:13.586532 13835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:27:13.591840 13962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:27:13.591913 13962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:27:13.594885 13692 master_runner.cc:263] No leader master found from master 8246b7e7ee264e38b9b7e2662cf20bd0
I20260619 02:27:13.594954 13692 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 02:27:13.595084 13713 tablet.cc:2369] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:27:13.596462 13767 master_runner.cc:263] No leader master found from master 25f81d5df4704d6099da624d9601a088
I20260619 02:27:13.596526 13767 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:27:13.597745 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } } }
I20260619 02:27:13.597862 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:27:13.598047 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } } }
I20260619 02:27:13.598169 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:27:13.601461 13963 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:27:13.601557 13963 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:27:13.684580 13925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:27:13.684787 13925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } }
I20260619 02:27:13.685886 13925 leader_election.cc:288] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8246b7e7ee264e38b9b7e2662cf20bd0 (127.0.98.62:39139), 25f81d5df4704d6099da624d9601a088 (127.0.98.61:46739)
I20260619 02:27:13.686790 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806bbb007c9642348fba2af16f5da24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25f81d5df4704d6099da624d9601a088" is_pre_election: true
I20260619 02:27:13.687127 13809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 806bbb007c9642348fba2af16f5da24f in term 0.
I20260619 02:27:13.688002 13734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806bbb007c9642348fba2af16f5da24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" is_pre_election: true
I20260619 02:27:13.688304 13734 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 806bbb007c9642348fba2af16f5da24f in term 0.
I20260619 02:27:13.688903 13858 leader_election.cc:302] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25f81d5df4704d6099da624d9601a088, 806bbb007c9642348fba2af16f5da24f; no voters:
I20260619 02:27:13.689642 13925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:27:13.689752 13925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:27:13.689814 13925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:13.691232 13925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } }
I20260619 02:27:13.691577 13925 leader_election.cc:288] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [CANDIDATE]: Term 1 election: Requested vote from peers 8246b7e7ee264e38b9b7e2662cf20bd0 (127.0.98.62:39139), 25f81d5df4704d6099da624d9601a088 (127.0.98.61:46739)
I20260619 02:27:13.692191 13734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806bbb007c9642348fba2af16f5da24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0"
I20260619 02:27:13.692324 13734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:13.694202 13734 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 806bbb007c9642348fba2af16f5da24f in term 1.
I20260619 02:27:13.694554 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806bbb007c9642348fba2af16f5da24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25f81d5df4704d6099da624d9601a088"
I20260619 02:27:13.694691 13809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:13.695945 13809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 806bbb007c9642348fba2af16f5da24f in term 1.
I20260619 02:27:13.696264 13858 leader_election.cc:302] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25f81d5df4704d6099da624d9601a088, 806bbb007c9642348fba2af16f5da24f; no voters:
I20260619 02:27:13.700183 13925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:27:13.700435 13925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [term 1 LEADER]: Becoming Leader. State: Replica: 806bbb007c9642348fba2af16f5da24f, State: Running, Role: LEADER
I20260619 02:27:13.700778 13925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } }
I20260619 02:27:13.702581 13925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806bbb007c9642348fba2af16f5da24f. Latest consensus state: current_term: 1 leader_uuid: "806bbb007c9642348fba2af16f5da24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } } }
I20260619 02:27:13.702689 13925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:13.705868 13968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:27:13.706848 13968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:27:13.711601 13809 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [term 1 FOLLOWER]: Refusing update from remote peer 806bbb007c9642348fba2af16f5da24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:27:13.712468 13966 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:27:13.718698 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806bbb007c9642348fba2af16f5da24f. Latest consensus state: current_term: 1 leader_uuid: "806bbb007c9642348fba2af16f5da24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } } }
I20260619 02:27:13.718824 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:27:13.720280 13734 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [term 1 FOLLOWER]: Refusing update from remote peer 806bbb007c9642348fba2af16f5da24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:27:13.720777 13966 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:27:13.724014 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806bbb007c9642348fba2af16f5da24f. Latest consensus state: current_term: 1 leader_uuid: "806bbb007c9642348fba2af16f5da24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } } }
I20260619 02:27:13.724129 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:27:13.729691 13966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "806bbb007c9642348fba2af16f5da24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } } }
I20260619 02:27:13.729802 13966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:13.730716 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "806bbb007c9642348fba2af16f5da24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } } }
I20260619 02:27:13.730823 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:27:13.731129 13969 mvcc.cc:205] Tried to move back new op lower bound from 7298400394077233152 to 7298400394041774080. Current Snapshot: MvccSnapshot[applied={T|T < 7298400394077233152}]
I20260619 02:27:13.731940 13925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "806bbb007c9642348fba2af16f5da24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } } }
I20260619 02:27:13.732041 13925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:13.732789 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "806bbb007c9642348fba2af16f5da24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8246b7e7ee264e38b9b7e2662cf20bd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } peers { permanent_uuid: "25f81d5df4704d6099da624d9601a088" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46739 } } peers { permanent_uuid: "806bbb007c9642348fba2af16f5da24f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45739 } } }
I20260619 02:27:13.732899 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:27:13.733566 13972 mvcc.cc:205] Tried to move back new op lower bound from 7298400394077233152 to 7298400394041774080. Current Snapshot: MvccSnapshot[applied={T|T < 7298400394077233152}]
I20260619 02:27:13.737077 13970 mvcc.cc:205] Tried to move back new op lower bound from 7298400394077233152 to 7298400394041774080. Current Snapshot: MvccSnapshot[applied={T|T < 7298400394077233152}]
I20260619 02:27:13.737299 13968 catalog_manager.cc:1347] Generated new cluster ID: e13d5cae95a14139b843c96ac5d5c044
I20260619 02:27:13.737361 13968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:27:13.763515 13968 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:27:13.764649 13968 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:27:13.777243 13968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 806bbb007c9642348fba2af16f5da24f: Generated new TSK 0
I20260619 02:27:13.778124 13968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:27:14.593282 13962 catalog_manager.cc:1259] Loaded cluster ID: e13d5cae95a14139b843c96ac5d5c044
I20260619 02:27:14.593405 13962 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0: loading cluster ID for follower catalog manager: success
I20260619 02:27:14.595319 13962 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0: acquiring CA information for follower catalog manager: success
I20260619 02:27:14.596566 13962 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8246b7e7ee264e38b9b7e2662cf20bd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:27:14.602654 13963 catalog_manager.cc:1259] Loaded cluster ID: e13d5cae95a14139b843c96ac5d5c044
I20260619 02:27:14.602748 13963 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088: loading cluster ID for follower catalog manager: success
I20260619 02:27:14.604367 13963 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088: acquiring CA information for follower catalog manager: success
I20260619 02:27:14.605458 13963 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25f81d5df4704d6099da624d9601a088: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:27:14.846345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 13692
I20260619 02:27:14.853114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 13767
I20260619 02:27:14.858829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 13841
2026-06-19T02:27:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3091 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:27:14 dist-test-slave-xkrt krb5kdc[13982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:27:14 dist-test-slave-xkrt krb5kdc[13982](info): set up 2 sockets
Jun 19 02:27:14 dist-test-slave-xkrt krb5kdc[13982](info): commencing operation
krb5kdc: starting...
W20260619 02:27:15.897761 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:27:15 dist-test-slave-xkrt krb5kdc[13982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:27:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:27:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:27:16.014948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45079
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:16.138758 13995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:16.139037 13995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:16.139076 13995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:16.143146 13995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:27:16.143206 13995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:16.143225 13995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:16.143241 13995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:27:16.143257 13995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:27:16.148561 13995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:16.149632 13995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:16.150632 13995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:16.158689 14000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:16.158824 14001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:16.158715 14003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:16.159034 13995 server_base.cc:1034] running on GCE node
I20260619 02:27:16.159492 13995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:16.160553 13995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:16.161703 13995 hybrid_clock.cc:648] HybridClock initialized: now 1781836036161680 us; error 37 us; skew 500 ppm
Jun 19 02:27:16 dist-test-slave-xkrt krb5kdc[13982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:16.164379 13995 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:27:16.165181 13995 webserver.cc:460] Webserver started at http://127.0.98.62:33733/ using document root <none> and password file <none>
I20260619 02:27:16.165766 13995 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:16.165828 13995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:16.166042 13995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:27:16.168013 13995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "24f514058f60410f87e39b4806e40b85"
format_stamp: "Formatted at 2026-06-19 02:27:16 on dist-test-slave-xkrt"
server_key: "ecefb5dfc749a674330525aa7eb3782a"
server_key_iv: "d0051e6a39d73bbfc4274b7f9d2d7133"
server_key_version: "encryptionkey@0"
I20260619 02:27:16.168546 13995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "24f514058f60410f87e39b4806e40b85"
format_stamp: "Formatted at 2026-06-19 02:27:16 on dist-test-slave-xkrt"
server_key: "ecefb5dfc749a674330525aa7eb3782a"
server_key_iv: "d0051e6a39d73bbfc4274b7f9d2d7133"
server_key_version: "encryptionkey@0"
I20260619 02:27:16.172086 13995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:27:16.174429 14010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:16.175587 13995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 02:27:16.175704 13995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "24f514058f60410f87e39b4806e40b85"
format_stamp: "Formatted at 2026-06-19 02:27:16 on dist-test-slave-xkrt"
server_key: "ecefb5dfc749a674330525aa7eb3782a"
server_key_iv: "d0051e6a39d73bbfc4274b7f9d2d7133"
server_key_version: "encryptionkey@0"
I20260619 02:27:16.175799 13995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:16.191668 13995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:16.194496 13995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:16.194685 13995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:16.201213 13995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45079
I20260619 02:27:16.201232 14062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45079 every 8 connection(s)
I20260619 02:27:16.202172 13995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:27:16.204833 14063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:16.210117 14063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85: Bootstrap starting.
I20260619 02:27:16.210813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 13995
I20260619 02:27:16.210940 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:27:16.211213 392 external_mini_cluster.cc:1428] Setting key c6c59ff5ed638c5e192f0f8054995200
I20260619 02:27:16.212898 14063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:16.213636 14063 log.cc:824] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:16.215680 14063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85: No bootstrap required, opened a new log
I20260619 02:27:16.219525 14063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f514058f60410f87e39b4806e40b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } }
I20260619 02:27:16.219774 14063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:16.219854 14063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f514058f60410f87e39b4806e40b85, State: Initialized, Role: FOLLOWER
Jun 19 02:27:16 dist-test-slave-xkrt krb5kdc[13982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:16.220381 14063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f514058f60410f87e39b4806e40b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } }
I20260619 02:27:16.220624 14063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:27:16.220715 14063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:27:16.220824 14063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:16.221764 14063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f514058f60410f87e39b4806e40b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } }
I20260619 02:27:16.222096 14063 leader_election.cc:302] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24f514058f60410f87e39b4806e40b85; no voters:
I20260619 02:27:16.222391 14063 leader_election.cc:288] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:27:16.222503 14068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:27:16.222707 14068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [term 1 LEADER]: Becoming Leader. State: Replica: 24f514058f60410f87e39b4806e40b85, State: Running, Role: LEADER
I20260619 02:27:16.223078 14068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f514058f60410f87e39b4806e40b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } }
I20260619 02:27:16.223778 14063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:27:16.224644 14070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24f514058f60410f87e39b4806e40b85. Latest consensus state: current_term: 1 leader_uuid: "24f514058f60410f87e39b4806e40b85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f514058f60410f87e39b4806e40b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } }
I20260619 02:27:16.224768 14070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:16.225164 14077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:27:16.226048 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24f514058f60410f87e39b4806e40b85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f514058f60410f87e39b4806e40b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } }
I20260619 02:27:16.226253 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:16.226929 14066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:16.212416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.0.98.62:45079)
0619 02:27:16.212796 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:16.212814 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:16.212856 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:27:16.213504 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:16.213520 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:16.213935 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:16.214207 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:16.214906 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:16.216331 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:16.217147 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:16.217879 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:16.218040 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:16.220656 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:16.220666 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:16.220686 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:16.220729 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:16.222802 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:16.223367 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:16.223373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:16.223378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:16.223514 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:16.223843 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:16.223846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:16.223848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:16.224362 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:16.224549 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:16.225479 (+ 930us) server_negotiation.cc:299] Negotiation successful
0619 02:27:16.225857 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":123,"threads_started":1}
I20260619 02:27:16.228282 14077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:27:16.230077 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:27:16.233533 14077 catalog_manager.cc:1347] Generated new cluster ID: 9df3f40b2c8c4a3d98408a7704303f2a
I20260619 02:27:16.233623 14077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:27:16.258113 14077 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:27:16.258999 14077 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:27:16.266188 14077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24f514058f60410f87e39b4806e40b85: Generated new TSK 0
I20260619 02:27:16.266839 14077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:27:16.286127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 13995
2026-06-19T02:27:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1425 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:27:16 dist-test-slave-xkrt krb5kdc[14090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:27:16 dist-test-slave-xkrt krb5kdc[14090](info): set up 2 sockets
Jun 19 02:27:16 dist-test-slave-xkrt krb5kdc[14090](info): commencing operation
krb5kdc: starting...
W20260619 02:27:17.321640 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:27:17 dist-test-slave-xkrt krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:27:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:27:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:27:17.438480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33401
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:17.543840 14103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:17.544076 14103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:17.544114 14103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:17.548007 14103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:27:17.548058 14103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:17.548076 14103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:17.548091 14103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:27:17.548106 14103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:27:17.553102 14103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:17.554036 14103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:17.554953 14103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:17.563064 14108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:17.563158 14103 server_base.cc:1034] running on GCE node
W20260619 02:27:17.563064 14111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:17.563102 14109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:17.563889 14103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:17.564946 14103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:17.566141 14103 hybrid_clock.cc:648] HybridClock initialized: now 1781836037566127 us; error 28 us; skew 500 ppm
Jun 19 02:27:17 dist-test-slave-xkrt krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:17.572722 14103 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:27:17.573655 14103 webserver.cc:460] Webserver started at http://127.0.98.62:35563/ using document root <none> and password file <none>
I20260619 02:27:17.574148 14103 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:17.574203 14103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:17.574364 14103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:27:17.576253 14103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "8af98ff02a1e401a93b1d9a2dd9c2033"
format_stamp: "Formatted at 2026-06-19 02:27:17 on dist-test-slave-xkrt"
server_key: "63bfcdebdbb27d3dd832b7faace22386"
server_key_iv: "deeeab22b52990723e5fd90d2b2154b7"
server_key_version: "encryptionkey@0"
I20260619 02:27:17.576740 14103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "8af98ff02a1e401a93b1d9a2dd9c2033"
format_stamp: "Formatted at 2026-06-19 02:27:17 on dist-test-slave-xkrt"
server_key: "63bfcdebdbb27d3dd832b7faace22386"
server_key_iv: "deeeab22b52990723e5fd90d2b2154b7"
server_key_version: "encryptionkey@0"
I20260619 02:27:17.580305 14103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:27:17.582859 14118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:17.584161 14103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:27:17.584272 14103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "8af98ff02a1e401a93b1d9a2dd9c2033"
format_stamp: "Formatted at 2026-06-19 02:27:17 on dist-test-slave-xkrt"
server_key: "63bfcdebdbb27d3dd832b7faace22386"
server_key_iv: "deeeab22b52990723e5fd90d2b2154b7"
server_key_version: "encryptionkey@0"
I20260619 02:27:17.584357 14103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:17.612350 14103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:17.615600 14103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:17.615749 14103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:17.622787 14103 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33401
I20260619 02:27:17.623076 14170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33401 every 8 connection(s)
I20260619 02:27:17.623737 14103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:27:17.626456 14171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:17.631805 14171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Bootstrap starting.
I20260619 02:27:17.634030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 14103
I20260619 02:27:17.634157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:27:17.634296 14171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:17.634476 392 external_mini_cluster.cc:1428] Setting key 4995e7c1f1985717f2189dd086c809ac
I20260619 02:27:17.635165 14171 log.cc:824] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:17.637337 14171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: No bootstrap required, opened a new log
I20260619 02:27:17.641618 14171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } }
I20260619 02:27:17.641855 14171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:17.641954 14171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af98ff02a1e401a93b1d9a2dd9c2033, State: Initialized, Role: FOLLOWER
I20260619 02:27:17.642395 14171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } }
I20260619 02:27:17.642517 14171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:27:17.642604 14171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:27:17.642719 14171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:17.643678 14171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } }
Jun 19 02:27:17 dist-test-slave-xkrt krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:17.644063 14171 leader_election.cc:302] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8af98ff02a1e401a93b1d9a2dd9c2033; no voters:
I20260619 02:27:17.644400 14171 leader_election.cc:288] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:27:17.644493 14176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:27:17.644739 14176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 1 LEADER]: Becoming Leader. State: Replica: 8af98ff02a1e401a93b1d9a2dd9c2033, State: Running, Role: LEADER
I20260619 02:27:17.645126 14176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } }
I20260619 02:27:17.645795 14171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:27:17.646934 14177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8af98ff02a1e401a93b1d9a2dd9c2033. Latest consensus state: current_term: 1 leader_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } } }
I20260619 02:27:17.647063 14177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:17.647439 14178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } } }
I20260619 02:27:17.647579 14178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:17.648214 14185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:27:17.650038 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:17.635599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.0.98.62:33401)
0619 02:27:17.636093 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:17.636112 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:17.636152 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:27:17.636955 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:17.636977 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:17.637356 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:17.637753 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:17.638357 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:17.639845 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:17.640719 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:17.641404 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:17.641582 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:17.644556 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:17.644564 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:17.644581 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:17.644615 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:17.646775 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:17.647324 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:17.647330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:17.647335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:17.647489 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:17.647842 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:17.647845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:17.647847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:17.648344 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:17.648545 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:17.648815 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 02:27:17.649136 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":164,"threads_started":1}
I20260619 02:27:17.650966 14185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:27:17.655907 14185 catalog_manager.cc:1347] Generated new cluster ID: a8e4dbb50f50422b84dd29e1825ef4e8
I20260619 02:27:17.655970 14185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:27:17.675272 14185 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:27:17.676457 14185 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:27:17.682992 14185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Generated new TSK 0
I20260619 02:27:17.683682 14185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:27:17.722296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33401
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:17.837180 14197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:17.837410 14197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:17.837476 14197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:17.841384 14197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:17.841439 14197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:17.841557 14197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:27:17.846691 14197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:17.847657 14197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:17.848606 14197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:17.857719 14205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:17.857753 14203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:17.857733 14202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:17.857999 14197 server_base.cc:1034] running on GCE node
I20260619 02:27:17.858362 14197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:17.858963 14197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:17.860121 14197 hybrid_clock.cc:648] HybridClock initialized: now 1781836037860102 us; error 30 us; skew 500 ppm
Jun 19 02:27:17 dist-test-slave-xkrt krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:17.862669 14197 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:27:17.863482 14197 webserver.cc:460] Webserver started at http://127.0.98.1:41935/ using document root <none> and password file <none>
I20260619 02:27:17.864002 14197 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:17.864060 14197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:17.864279 14197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:27:17.866212 14197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "d1632bcf8846476c8b09814f67ec7753"
format_stamp: "Formatted at 2026-06-19 02:27:17 on dist-test-slave-xkrt"
server_key: "f00f681e3f669f9b3bfc38dc9031e3e2"
server_key_iv: "9fe0853058962c3a016295f0b5a4798b"
server_key_version: "encryptionkey@0"
I20260619 02:27:17.866729 14197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d1632bcf8846476c8b09814f67ec7753"
format_stamp: "Formatted at 2026-06-19 02:27:17 on dist-test-slave-xkrt"
server_key: "f00f681e3f669f9b3bfc38dc9031e3e2"
server_key_iv: "9fe0853058962c3a016295f0b5a4798b"
server_key_version: "encryptionkey@0"
I20260619 02:27:17.870190 14197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:27:17.872306 14212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:17.873456 14197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:27:17.873595 14197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "d1632bcf8846476c8b09814f67ec7753"
format_stamp: "Formatted at 2026-06-19 02:27:17 on dist-test-slave-xkrt"
server_key: "f00f681e3f669f9b3bfc38dc9031e3e2"
server_key_iv: "9fe0853058962c3a016295f0b5a4798b"
server_key_version: "encryptionkey@0"
I20260619 02:27:17.873697 14197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:17.888535 14197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:17.891168 14197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:17.891362 14197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:17.891891 14197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:17.892642 14197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:17.892709 14197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:17.892771 14197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:17.892810 14197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:17.901415 14197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39357
I20260619 02:27:17.901445 14325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39357 every 8 connection(s)
I20260619 02:27:17.902266 14197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:27:17.908351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 14197
I20260619 02:27:17.908511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:27:17.908792 392 external_mini_cluster.cc:1428] Setting key da254234154cb5b111d612f6ba1bc9c8
Jun 19 02:27:17 dist-test-slave-xkrt krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:17.915297 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:17.903935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57683 (local address 127.0.98.62:33401)
0619 02:27:17.904055 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:17.904059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:17.904964 (+ 905us) server_negotiation.cc:407] Connection header received
0619 02:27:17.905899 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:17.905903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:17.905991 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:17.906080 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:17.907167 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:17.907713 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:17.908963 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:17.909196 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:17.909300 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:17.911838 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:17.911846 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:17.911848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:17.911871 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:17.913259 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:17.913834 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:17.913839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:17.913841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:17.913922 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:17.914252 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:17.914256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:17.914258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:17.914435 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:17.914538 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:17.915011 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 02:27:17.915129 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:27:17.916090 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:17.904219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:57683)
0619 02:27:17.904819 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:17.904857 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:17.905708 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:17.906241 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:17.906248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:17.906577 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:17.907027 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:17.907040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:17.907833 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:17.907841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:17.908824 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:17.908833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:17.909347 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:17.909351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:17.909589 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:17.910236 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:17.910256 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:17.911685 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:17.913391 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:17.913398 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:17.913412 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:17.913736 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:17.914013 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:17.914017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:17.914019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:17.914156 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:17.914537 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:17.914543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:17.914777 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 02:27:17.914956 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 02:27:17.915257 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":105,"threads_started":1}
I20260619 02:27:17.917228 14326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33401
I20260619 02:27:17.917475 14326 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:27:17.918078 14326 heartbeater.cc:511] Master 127.0.98.62:33401 requested a full tablet report, sending...
I20260619 02:27:17.919493 14135 ts_manager.cc:194] Registered new tserver with Master: d1632bcf8846476c8b09814f67ec7753 (127.0.98.1:39357)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 02:27:17.920799 14135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:27:17.936472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33401
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:18.044312 14331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:18.044564 14331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:18.044606 14331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:18.048772 14331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:18.048831 14331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:18.048887 14331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:27:18.054055 14331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:18.055116 14331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:18.056056 14331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:18.065361 14339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:18.065376 14337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:18.065408 14336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:18.065575 14331 server_base.cc:1034] running on GCE node
I20260619 02:27:18.066116 14331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:18.066707 14331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:18.067873 14331 hybrid_clock.cc:648] HybridClock initialized: now 1781836038067847 us; error 26 us; skew 500 ppm
Jun 19 02:27:18 dist-test-slave-xkrt krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:18.070434 14331 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:27:18.071285 14331 webserver.cc:460] Webserver started at http://127.0.98.2:43645/ using document root <none> and password file <none>
I20260619 02:27:18.071818 14331 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:18.071882 14331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:18.072098 14331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:27:18.074149 14331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "b15a9351deb148ee9c9a1983b07eb3c4"
format_stamp: "Formatted at 2026-06-19 02:27:18 on dist-test-slave-xkrt"
server_key: "a4a446e4db00dc9bad9e994220c3c6a0"
server_key_iv: "15845a0951b1b5943cdb74fe47d70a37"
server_key_version: "encryptionkey@0"
I20260619 02:27:18.074707 14331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b15a9351deb148ee9c9a1983b07eb3c4"
format_stamp: "Formatted at 2026-06-19 02:27:18 on dist-test-slave-xkrt"
server_key: "a4a446e4db00dc9bad9e994220c3c6a0"
server_key_iv: "15845a0951b1b5943cdb74fe47d70a37"
server_key_version: "encryptionkey@0"
I20260619 02:27:18.078545 14331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:27:18.080850 14346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:18.082088 14331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:27:18.082211 14331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "b15a9351deb148ee9c9a1983b07eb3c4"
format_stamp: "Formatted at 2026-06-19 02:27:18 on dist-test-slave-xkrt"
server_key: "a4a446e4db00dc9bad9e994220c3c6a0"
server_key_iv: "15845a0951b1b5943cdb74fe47d70a37"
server_key_version: "encryptionkey@0"
I20260619 02:27:18.082311 14331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:18.093477 14331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:18.096129 14331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:18.096307 14331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:18.096823 14331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:18.097625 14331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:18.097685 14331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:18.097746 14331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:18.097790 14331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:18.106294 14331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38681
I20260619 02:27:18.106328 14459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38681 every 8 connection(s)
I20260619 02:27:18.107103 14331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:27:18.112239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 14331
I20260619 02:27:18.112361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:27:18.112629 392 external_mini_cluster.cc:1428] Setting key 8e8e6ccef12af6b187b4b3680ae9ec8a
Jun 19 02:27:18 dist-test-slave-xkrt krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:18.120749 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.108888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55405 (local address 127.0.98.62:33401)
0619 02:27:18.109012 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:18.109017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:18.110041 (+ 1024us) server_negotiation.cc:407] Connection header received
0619 02:27:18.110878 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:18.110881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:18.110928 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:18.111004 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:18.112158 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.112822 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.114203 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.114426 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.114504 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.117358 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:18.117363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:18.117365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:18.117387 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:18.118822 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:18.119319 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:18.119327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:18.119328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:18.119400 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:18.119714 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:18.119719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:18.119720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:18.119902 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:18.120002 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:18.120465 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 02:27:18.120586 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:27:18.121510 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.109221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:55405)
0619 02:27:18.109893 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:18.109931 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:18.110656 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:18.111161 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:18.111168 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:18.111510 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:18.111959 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:18.111996 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.112950 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.112954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:18.114094 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:18.114105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.114570 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.114576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:18.114796 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.115367 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:18.115391 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:18.117204 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:18.118950 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:18.118955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:18.118968 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:18.119215 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:18.119490 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:18.119494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:18.119496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:18.119627 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:18.120001 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:18.120007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:18.120241 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 02:27:18.120425 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 02:27:18.120726 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:27:18.122712 14460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33401
I20260619 02:27:18.122960 14460 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:18.123443 14460 heartbeater.cc:511] Master 127.0.98.62:33401 requested a full tablet report, sending...
I20260619 02:27:18.124465 14135 ts_manager.cc:194] Registered new tserver with Master: b15a9351deb148ee9c9a1983b07eb3c4 (127.0.98.2:38681)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 02:27:18.125262 14135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:27:18.138764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33401
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:18.246284 14465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:18.246548 14465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:18.246618 14465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:18.250725 14465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:18.250802 14465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:18.250892 14465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:27:18.255882 14465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:18.256860 14465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:18.257853 14465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:18.266968 14471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:18.266973 14470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:18.267037 14473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:18.267135 14465 server_base.cc:1034] running on GCE node
I20260619 02:27:18.267594 14465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:18.268229 14465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:18.269405 14465 hybrid_clock.cc:648] HybridClock initialized: now 1781836038269389 us; error 29 us; skew 500 ppm
Jun 19 02:27:18 dist-test-slave-xkrt krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:18.271996 14465 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:27:18.272789 14465 webserver.cc:460] Webserver started at http://127.0.98.3:41027/ using document root <none> and password file <none>
I20260619 02:27:18.273325 14465 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:18.273424 14465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:18.273689 14465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:27:18.275651 14465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "f4441ae3c81945b7a028c6903460e4f0"
format_stamp: "Formatted at 2026-06-19 02:27:18 on dist-test-slave-xkrt"
server_key: "b11a40103cbfb6cea1b70345ef6df567"
server_key_iv: "76f5ad34aac31395f35df374ee9e7021"
server_key_version: "encryptionkey@0"
I20260619 02:27:18.276194 14465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f4441ae3c81945b7a028c6903460e4f0"
format_stamp: "Formatted at 2026-06-19 02:27:18 on dist-test-slave-xkrt"
server_key: "b11a40103cbfb6cea1b70345ef6df567"
server_key_iv: "76f5ad34aac31395f35df374ee9e7021"
server_key_version: "encryptionkey@0"
I20260619 02:27:18.279844 14465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 02:27:18.282061 14480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:18.283178 14465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:27:18.283308 14465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "f4441ae3c81945b7a028c6903460e4f0"
format_stamp: "Formatted at 2026-06-19 02:27:18 on dist-test-slave-xkrt"
server_key: "b11a40103cbfb6cea1b70345ef6df567"
server_key_iv: "76f5ad34aac31395f35df374ee9e7021"
server_key_version: "encryptionkey@0"
I20260619 02:27:18.283418 14465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:18.317821 14465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:18.320689 14465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:18.320907 14465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:18.321498 14465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:18.322322 14465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:18.322394 14465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:18.322461 14465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:18.322508 14465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:18.331178 14465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41505
I20260619 02:27:18.331208 14593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41505 every 8 connection(s)
I20260619 02:27:18.332026 14465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:27:18.334312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 14465
I20260619 02:27:18.334434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:27:18.334713 392 external_mini_cluster.cc:1428] Setting key 9b306a3a16959ce48b9d296fc547df4d
Jun 19 02:27:18 dist-test-slave-xkrt krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:18.344130 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.333656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43079 (local address 127.0.98.62:33401)
0619 02:27:18.333778 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:18.333782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:18.334904 (+ 1122us) server_negotiation.cc:407] Connection header received
0619 02:27:18.335910 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:18.335914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:18.335972 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:18.336036 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:18.337250 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.337792 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.338832 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.339118 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.339199 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.341305 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:18.341308 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:18.341311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:18.341333 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:18.342534 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:18.342961 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:18.342963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:18.342964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:18.343032 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:18.343290 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:18.343292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:18.343293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:18.343449 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:18.343520 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:18.343928 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 02:27:18.344017 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:27:18.344851 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.333998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:43079)
0619 02:27:18.334721 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:18.334778 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:18.335681 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:18.336218 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:18.336225 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:18.336606 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:18.337129 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:18.337142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.337892 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.337895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:18.338735 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:18.338743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.339240 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.339243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:18.339382 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.339857 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:18.339874 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:18.341170 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:18.342636 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:18.342641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:18.342651 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:18.342882 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:18.343103 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:18.343105 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:18.343107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:18.343220 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:18.343526 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:18.343530 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:18.343738 (+ 208us) client_negotiation.cc:769] Sending connection context
0619 02:27:18.343882 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 02:27:18.344099 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":106,"threads_started":1}
I20260619 02:27:18.345897 14594 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33401
I20260619 02:27:18.346120 14594 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:18.346593 14594 heartbeater.cc:511] Master 127.0.98.62:33401 requested a full tablet report, sending...
I20260619 02:27:18.347713 14134 ts_manager.cc:194] Registered new tserver with Master: f4441ae3c81945b7a028c6903460e4f0 (127.0.98.3:41505)
I20260619 02:27:18.348332 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:27:18.348433 14134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43079
I20260619 02:27:18.357923 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.350344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.0.98.62:33401)
0619 02:27:18.350478 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:18.350482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:18.350647 (+ 165us) server_negotiation.cc:407] Connection header received
0619 02:27:18.350739 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:18.350741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:18.350774 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:18.350830 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:18.351416 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.351895 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.352651 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.352884 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.353014 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.354578 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:18.354582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:18.354585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:18.354612 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:18.356389 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:18.356821 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:18.356824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:18.356825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:18.356900 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:18.357075 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:18.357077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:18.357078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:18.357270 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:18.357384 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:18.357589 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 02:27:18.357697 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:27:18.360644 14134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:27:18.363879 14134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:27:18.378067 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.371717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38681 (local address 127.0.0.1:47598)
0619 02:27:18.372693 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:18.372751 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:18.372882 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:18.373580 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:18.373586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:18.373603 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:18.373728 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:18.373739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.375240 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.375243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:18.376308 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:18.376317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.377322 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.377326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:18.377453 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.377496 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 02:27:18.377620 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 02:27:18.377736 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":30,"spinlock_wait_cycles":5376,"thread_start_us":65,"threads_started":1}
I20260619 02:27:18.378096 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.372583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.0.98.3:41505)
0619 02:27:18.373003 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:18.373016 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:18.373040 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:27:18.373113 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:18.373118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:18.373326 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:18.373486 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:18.373841 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.375055 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.376675 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.377459 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.377600 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.377724 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:18.377876 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 02:27:18.377970 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":118,"threads_started":1}
I20260619 02:27:18.378265 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.371880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.0.98.2:38681)
0619 02:27:18.373147 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:18.373158 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:18.373175 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:27:18.373248 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:18.373252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:18.373403 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:18.373900 (+ 497us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:18.374054 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.375126 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.376426 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.377199 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.377311 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.377452 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:18.377980 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 02:27:18.378142 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1176,"thread_start_us":125,"threads_started":1}
I20260619 02:27:18.378082 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.371905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41505 (local address 127.0.0.1:56704)
0619 02:27:18.372701 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:18.372747 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:18.372882 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:18.373484 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:18.373490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:18.373513 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:18.373719 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:18.373726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.375185 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.375189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:18.376545 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:18.376558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.377604 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.377607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:18.377700 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.377721 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:27:18.377812 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 02:27:18.377881 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":217,"spinlock_wait_cycles":8064,"thread_start_us":106,"threads_started":1}
I20260619 02:27:18.378638 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.372111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39357 (local address 127.0.0.1:51220)
0619 02:27:18.373086 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:18.373100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:18.373169 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:18.374006 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:18.374008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:18.374022 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:18.374077 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:18.374081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.375051 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.375054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:18.376566 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:18.376572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.377685 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.377688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:18.377783 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.377804 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:27:18.377874 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:27:18.377948 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":47,"thread_start_us":75,"threads_started":1}
I20260619 02:27:18.380745 14528 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3ccf3ffb784ba58b2415a22a3d4281 (DEFAULT_TABLE table=test-table [id=01b5f06f43cf4906ad02c3b144b08774]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:18.380743 14394 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3ccf3ffb784ba58b2415a22a3d4281 (DEFAULT_TABLE table=test-table [id=01b5f06f43cf4906ad02c3b144b08774]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:18.381385 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.372465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.0.98.1:39357)
0619 02:27:18.373389 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:18.373401 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:18.373441 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:27:18.373509 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:18.373514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:18.373728 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:18.373863 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:18.374182 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.374924 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.376701 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.377378 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.377476 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.377631 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:18.379003 (+ 1372us) server_negotiation.cc:299] Negotiation successful
0619 02:27:18.379099 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":69,"threads_started":1}
I20260619 02:27:18.381731 14394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3ccf3ffb784ba58b2415a22a3d4281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:18.381747 14528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3ccf3ffb784ba58b2415a22a3d4281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:18.383911 14260 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3ccf3ffb784ba58b2415a22a3d4281 (DEFAULT_TABLE table=test-table [id=01b5f06f43cf4906ad02c3b144b08774]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:18.384796 14260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3ccf3ffb784ba58b2415a22a3d4281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:18.388072 14611 tablet_bootstrap.cc:492] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0: Bootstrap starting.
I20260619 02:27:18.389287 14612 tablet_bootstrap.cc:492] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4: Bootstrap starting.
I20260619 02:27:18.390209 14611 tablet_bootstrap.cc:654] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:18.391278 14611 log.cc:824] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:18.391438 14613 tablet_bootstrap.cc:492] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Bootstrap starting.
I20260619 02:27:18.391772 14612 tablet_bootstrap.cc:654] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:18.392438 14612 log.cc:824] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:18.393298 14613 tablet_bootstrap.cc:654] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:18.394436 14611 tablet_bootstrap.cc:492] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0: No bootstrap required, opened a new log
I20260619 02:27:18.394493 14612 tablet_bootstrap.cc:492] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4: No bootstrap required, opened a new log
I20260619 02:27:18.394541 14613 log.cc:824] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:18.394659 14612 ts_tablet_manager.cc:1397] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:27:18.394655 14611 ts_tablet_manager.cc:1397] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 02:27:18.396246 14613 tablet_bootstrap.cc:492] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: No bootstrap required, opened a new log
I20260619 02:27:18.396409 14613 ts_tablet_manager.cc:1397] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:27:18.397630 14611 raft_consensus.cc:348] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:18.397864 14611 raft_consensus.cc:374] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:18.397938 14611 raft_consensus.cc:729] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4441ae3c81945b7a028c6903460e4f0, State: Initialized, Role: FOLLOWER
I20260619 02:27:18.398350 14611 consensus_queue.cc:260] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:18.398339 14612 raft_consensus.cc:348] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:18.398613 14612 raft_consensus.cc:374] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:18.398706 14612 raft_consensus.cc:729] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15a9351deb148ee9c9a1983b07eb3c4, State: Initialized, Role: FOLLOWER
I20260619 02:27:18.398917 14594 heartbeater.cc:503] Master 127.0.98.62:33401 was elected leader, sending a full tablet report...
I20260619 02:27:18.399200 14612 consensus_queue.cc:260] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:18.400003 14611 ts_tablet_manager.cc:1428] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:27:18.400058 14612 ts_tablet_manager.cc:1428] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:27:18.399997 14613 raft_consensus.cc:348] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:18.400286 14613 raft_consensus.cc:374] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:18.400384 14613 raft_consensus.cc:729] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1632bcf8846476c8b09814f67ec7753, State: Initialized, Role: FOLLOWER
I20260619 02:27:18.400403 14460 heartbeater.cc:503] Master 127.0.98.62:33401 was elected leader, sending a full tablet report...
I20260619 02:27:18.400767 14613 consensus_queue.cc:260] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:18.401355 14326 heartbeater.cc:503] Master 127.0.98.62:33401 was elected leader, sending a full tablet report...
I20260619 02:27:18.401788 14613 ts_tablet_manager.cc:1428] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 02:27:18.403458 14327 tablet.cc:2369] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:27:18.582926 14595 tablet.cc:2369] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:27:18.608470 14461 tablet.cc:2369] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:27:18.692291 14618 raft_consensus.cc:482] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:27:18.692509 14618 raft_consensus.cc:504] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:18.693745 14618 leader_election.cc:288] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4441ae3c81945b7a028c6903460e4f0 (127.0.98.3:41505), d1632bcf8846476c8b09814f67ec7753 (127.0.98.1:39357)
I20260619 02:27:18.697492 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.694082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34837 (local address 127.0.98.1:39357)
0619 02:27:18.694298 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:18.694305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:18.694479 (+ 174us) server_negotiation.cc:407] Connection header received
0619 02:27:18.694593 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:18.694595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:18.694662 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:18.694739 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:18.695036 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.695510 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.696576 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.696951 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.697058 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.697120 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:18.697279 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 02:27:18.697356 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 02:27:18.697492 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.694007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39357 (local address 127.0.98.2:34837)
0619 02:27:18.694370 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:18.694385 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:18.694501 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:18.694761 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:18.694763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:18.694776 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:18.694933 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:18.694939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.695636 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.695639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:18.696463 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:18.696471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.697062 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.697066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:18.697161 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.697183 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:27:18.697248 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:27:18.697326 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":40,"thread_start_us":80,"threads_started":1}
I20260619 02:27:18.697517 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.694067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36779 (local address 127.0.98.3:41505)
0619 02:27:18.694235 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:18.694241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:18.694351 (+ 110us) server_negotiation.cc:407] Connection header received
0619 02:27:18.694494 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:18.694497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:18.694555 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:18.694632 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:18.694948 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.695476 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.696572 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.696964 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.697079 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.697139 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:18.697338 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 02:27:18.697413 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:27:18.697513 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.693935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41505 (local address 127.0.98.2:36779)
0619 02:27:18.694248 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:18.694263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:18.694392 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:18.694655 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:18.694658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:18.694673 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:18.694832 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:18.694840 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.695582 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.695587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:18.696456 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:18.696465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.697099 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.697102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:18.697208 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.697225 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:27:18.697314 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 02:27:18.697373 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260619 02:27:18.698194 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3ccf3ffb784ba58b2415a22a3d4281" candidate_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1632bcf8846476c8b09814f67ec7753" is_pre_election: true
I20260619 02:27:18.698343 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3ccf3ffb784ba58b2415a22a3d4281" candidate_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4441ae3c81945b7a028c6903460e4f0" is_pre_election: true
I20260619 02:27:18.698504 14280 raft_consensus.cc:2454] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15a9351deb148ee9c9a1983b07eb3c4 in term 0.
I20260619 02:27:18.698589 14548 raft_consensus.cc:2454] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15a9351deb148ee9c9a1983b07eb3c4 in term 0.
I20260619 02:27:18.699010 14347 leader_election.cc:302] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b15a9351deb148ee9c9a1983b07eb3c4, d1632bcf8846476c8b09814f67ec7753, f4441ae3c81945b7a028c6903460e4f0; no voters:
I20260619 02:27:18.699271 14618 raft_consensus.cc:2781] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:27:18.699345 14618 raft_consensus.cc:482] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:27:18.699411 14618 raft_consensus.cc:3037] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:18.700456 14618 raft_consensus.cc:504] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:18.700794 14618 leader_election.cc:288] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [CANDIDATE]: Term 1 election: Requested vote from peers f4441ae3c81945b7a028c6903460e4f0 (127.0.98.3:41505), d1632bcf8846476c8b09814f67ec7753 (127.0.98.1:39357)
I20260619 02:27:18.701109 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3ccf3ffb784ba58b2415a22a3d4281" candidate_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4441ae3c81945b7a028c6903460e4f0"
I20260619 02:27:18.701138 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3ccf3ffb784ba58b2415a22a3d4281" candidate_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1632bcf8846476c8b09814f67ec7753"
I20260619 02:27:18.701263 14548 raft_consensus.cc:3037] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:18.701270 14280 raft_consensus.cc:3037] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:18.702312 14548 raft_consensus.cc:2454] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15a9351deb148ee9c9a1983b07eb3c4 in term 1.
I20260619 02:27:18.702322 14280 raft_consensus.cc:2454] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15a9351deb148ee9c9a1983b07eb3c4 in term 1.
I20260619 02:27:18.702572 14347 leader_election.cc:302] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b15a9351deb148ee9c9a1983b07eb3c4, f4441ae3c81945b7a028c6903460e4f0; no voters:
I20260619 02:27:18.702729 14618 raft_consensus.cc:2781] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:27:18.702935 14618 raft_consensus.cc:686] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [term 1 LEADER]: Becoming Leader. State: Replica: b15a9351deb148ee9c9a1983b07eb3c4, State: Running, Role: LEADER
I20260619 02:27:18.703258 14618 consensus_queue.cc:237] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:18.706173 14135 catalog_manager.cc:5495] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b15a9351deb148ee9c9a1983b07eb3c4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } health_report { overall_health: UNKNOWN } } }
I20260619 02:27:18.735489 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.732384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.0.98.2:38681)
0619 02:27:18.732502 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:18.732505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:18.732517 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:27:18.732615 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:18.732617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:18.732668 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:18.732751 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:27:18.732935 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.733412 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.734325 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:18.734527 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:18.734599 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:18.734827 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:27:18.735202 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:27:18.735285 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:18.735335 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 02:27:18.735388 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:27:18.741070 14280 raft_consensus.cc:1261] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753 [term 1 FOLLOWER]: Refusing update from remote peer b15a9351deb148ee9c9a1983b07eb3c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:27:18.741070 14548 raft_consensus.cc:1261] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0 [term 1 FOLLOWER]: Refusing update from remote peer b15a9351deb148ee9c9a1983b07eb3c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:27:18.741813 14622 consensus_queue.cc:1035] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:27:18.741977 14618 consensus_queue.cc:1035] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:27:18.748613 14625 mvcc.cc:205] Tried to move back new op lower bound from 7298400414676258816 to 7298400414530850816. Current Snapshot: MvccSnapshot[applied={T|T < 7298400414676258816}]
I20260619 02:27:18.749145 14627 mvcc.cc:205] Tried to move back new op lower bound from 7298400414676258816 to 7298400414530850816. Current Snapshot: MvccSnapshot[applied={T|T < 7298400414676258816}]
I20260619 02:27:18.749625 14626 mvcc.cc:205] Tried to move back new op lower bound from 7298400414676258816 to 7298400414530850816. Current Snapshot: MvccSnapshot[applied={T|T < 7298400414676258816}]
I20260619 02:27:18.758257 14135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:27:18.758419 14135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:27:18.760557 14135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Sending DeleteTablet for 3 replicas of tablet 2d3ccf3ffb784ba58b2415a22a3d4281
I20260619 02:27:18.761241 14260 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3ccf3ffb784ba58b2415a22a3d4281 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51220
I20260619 02:27:18.761224 14394 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3ccf3ffb784ba58b2415a22a3d4281 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47598
I20260619 02:27:18.761282 14528 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3ccf3ffb784ba58b2415a22a3d4281 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56704
I20260619 02:27:18.761705 14635 tablet_replica.cc:330] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: stopping tablet replica
I20260619 02:27:18.761740 14637 tablet_replica.cc:330] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0: stopping tablet replica
I20260619 02:27:18.762001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 14197
I20260619 02:27:18.762071 14637 raft_consensus.cc:2229] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:27:18.762077 14635 raft_consensus.cc:2229] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:27:18.762344 14637 raft_consensus.cc:2258] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:27:18.763567 14636 tablet_replica.cc:330] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4: stopping tablet replica
I20260619 02:27:18.763840 14636 raft_consensus.cc:2229] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:27:18.764178 14637 ts_tablet_manager.cc:1905] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:27:18.764207 14636 raft_consensus.cc:2258] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:27:18.765586 14636 ts_tablet_manager.cc:1905] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:27:18.766871 14637 ts_tablet_manager.cc:1918] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:27:18.767015 14637 log.cc:1196] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/2d3ccf3ffb784ba58b2415a22a3d4281
I20260619 02:27:18.767407 14637 ts_tablet_manager.cc:1939] T 2d3ccf3ffb784ba58b2415a22a3d4281 P f4441ae3c81945b7a028c6903460e4f0: Deleting consensus metadata
I20260619 02:27:18.768620 14119 catalog_manager.cc:4841] TS f4441ae3c81945b7a028c6903460e4f0 (127.0.98.3:41505): tablet 2d3ccf3ffb784ba58b2415a22a3d4281 (table test-table [id=01b5f06f43cf4906ad02c3b144b08774]) successfully deleted
I20260619 02:27:18.768888 14636 ts_tablet_manager.cc:1918] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:27:18.768988 14636 log.cc:1196] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/2d3ccf3ffb784ba58b2415a22a3d4281
I20260619 02:27:18.769279 14636 ts_tablet_manager.cc:1939] T 2d3ccf3ffb784ba58b2415a22a3d4281 P b15a9351deb148ee9c9a1983b07eb3c4: Deleting consensus metadata
W20260619 02:27:18.770352 14120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39357 (error 108)
I20260619 02:27:18.770356 14121 catalog_manager.cc:4841] TS b15a9351deb148ee9c9a1983b07eb3c4 (127.0.98.2:38681): tablet 2d3ccf3ffb784ba58b2415a22a3d4281 (table test-table [id=01b5f06f43cf4906ad02c3b144b08774]) successfully deleted
I20260619 02:27:18.770601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 14331
I20260619 02:27:18.772554 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:18.772306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39357 (local address 127.0.0.1:51222)
0619 02:27:18.772431 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:18.772489 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 02:27:18.772753 14120 catalog_manager.cc:4584] TS d1632bcf8846476c8b09814f67ec7753 (127.0.98.1:39357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d3ccf3ffb784ba58b2415a22a3d4281: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39357: connect: Connection refused (error 111)
I20260619 02:27:18.776494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 14465
I20260619 02:27:18.781829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 14103
I20260619 02:27:18.787842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33401
--webserver_interface=127.0.98.62
--webserver_port=35563
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:18.894328 14639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:18.894575 14639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:18.894655 14639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:18.898559 14639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:27:18.898607 14639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:18.898626 14639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:18.898669 14639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:27:18.898703 14639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:27:18.903694 14639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:18.904645 14639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:18.905632 14639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:18.917779 14648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:18.917791 14646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:18.917789 14645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:18.918093 14639 server_base.cc:1034] running on GCE node
I20260619 02:27:18.918686 14639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:18.919818 14639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:18.921006 14639 hybrid_clock.cc:648] HybridClock initialized: now 1781836038920991 us; error 34 us; skew 500 ppm
Jun 19 02:27:18 dist-test-slave-xkrt krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:18.923614 14639 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:27:18.924485 14639 webserver.cc:460] Webserver started at http://127.0.98.62:35563/ using document root <none> and password file <none>
I20260619 02:27:18.925034 14639 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:18.925136 14639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:18.928803 14639 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:27:18.930938 14655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:18.932163 14639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:27:18.932258 14639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "8af98ff02a1e401a93b1d9a2dd9c2033"
format_stamp: "Formatted at 2026-06-19 02:27:17 on dist-test-slave-xkrt"
server_key: "63bfcdebdbb27d3dd832b7faace22386"
server_key_iv: "deeeab22b52990723e5fd90d2b2154b7"
server_key_version: "encryptionkey@0"
I20260619 02:27:18.932821 14639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:18.960829 14639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:18.963969 14639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:18.964123 14639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:18.970604 14639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33401
I20260619 02:27:18.970886 14707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33401 every 8 connection(s)
I20260619 02:27:18.971657 14639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:27:18.973217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 14639
I20260619 02:27:18.973672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39357
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33401
--webserver_port=41935
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:27:18.976524 14708 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:27:18.979092 14708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Bootstrap starting.
I20260619 02:27:19.001789 14708 log.cc:824] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:19.012064 14708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:27:19.012488 14708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Bootstrap complete.
I20260619 02:27:19.017987 14708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } }
I20260619 02:27:19.018477 14708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af98ff02a1e401a93b1d9a2dd9c2033, State: Initialized, Role: FOLLOWER
I20260619 02:27:19.019006 14708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } }
I20260619 02:27:19.019155 14708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:27:19.019227 14708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:27:19.019318 14708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:27:19.021720 14708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } }
I20260619 02:27:19.022089 14708 leader_election.cc:302] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8af98ff02a1e401a93b1d9a2dd9c2033; no voters:
I20260619 02:27:19.022423 14708 leader_election.cc:288] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:27:19.022533 14712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:27:19.022796 14712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 2 LEADER]: Becoming Leader. State: Replica: 8af98ff02a1e401a93b1d9a2dd9c2033, State: Running, Role: LEADER
I20260619 02:27:19.023108 14712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } }
I20260619 02:27:19.023607 14708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:27:19.024003 14714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8af98ff02a1e401a93b1d9a2dd9c2033. Latest consensus state: current_term: 2 leader_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } } }
I20260619 02:27:19.024124 14714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:19.024399 14713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } } }
I20260619 02:27:19.024477 14713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:19.024526 14716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:27:19.028759 14716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:27:19.029667 14716 catalog_manager.cc:1259] Loaded cluster ID: a8e4dbb50f50422b84dd29e1825ef4e8
I20260619 02:27:19.029726 14716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:27:19.030771 14716 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 02:27:19.033499 14716 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 02:27:19.038040 14716 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:27:19.038609 14716 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Loaded TSK: 0
I20260619 02:27:19.039507 14716 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:27:19.111127 14710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:19.111367 14710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:19.111407 14710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:19.115586 14710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:19.115638 14710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:19.115720 14710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:27:19.120791 14710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41935
--tserver_master_addrs=127.0.98.62:33401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:19.121807 14710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:19.122771 14710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:19.132313 14736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:19.132290 14735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:19.132330 14738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:19.132871 14710 server_base.cc:1034] running on GCE node
I20260619 02:27:19.133258 14710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:19.133930 14710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:19.135111 14710 hybrid_clock.cc:648] HybridClock initialized: now 1781836039135077 us; error 52 us; skew 500 ppm
Jun 19 02:27:19 dist-test-slave-xkrt krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:19.138263 14710 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:27:19.139124 14710 webserver.cc:460] Webserver started at http://127.0.98.1:41935/ using document root <none> and password file <none>
I20260619 02:27:19.139654 14710 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:19.139758 14710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:19.143600 14710 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:27:19.145763 14745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:19.147027 14710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:27:19.147176 14710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "d1632bcf8846476c8b09814f67ec7753"
format_stamp: "Formatted at 2026-06-19 02:27:17 on dist-test-slave-xkrt"
server_key: "f00f681e3f669f9b3bfc38dc9031e3e2"
server_key_iv: "9fe0853058962c3a016295f0b5a4798b"
server_key_version: "encryptionkey@0"
I20260619 02:27:19.147702 14710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:19.159610 14710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:19.162429 14710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:19.162616 14710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:19.163203 14710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:19.164233 14752 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:27:19.168157 14710 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:27:19.168215 14710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 02:27:19.168279 14710 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:27:19.169950 14710 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:27:19.170006 14710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 02:27:19.170258 14752 tablet_bootstrap.cc:492] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Bootstrap starting.
I20260619 02:27:19.180817 14710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39357
I20260619 02:27:19.180999 14859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39357 every 8 connection(s)
I20260619 02:27:19.182034 14710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:27:19.190429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 14710
I20260619 02:27:19.190891 14752 log.cc:824] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:19.190944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38681
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33401
--webserver_port=43645
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:27:19 dist-test-slave-xkrt krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:19.199704 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.184216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60059 (local address 127.0.98.62:33401)
0619 02:27:19.184607 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:19.184626 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:19.184911 (+ 285us) server_negotiation.cc:407] Connection header received
0619 02:27:19.186070 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:19.186080 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:19.186366 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:19.186684 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:19.187787 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.188730 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.189942 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.190571 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.190756 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.193684 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:19.193690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:19.193753 (+ 63us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:19.193782 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:19.195947 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:19.196489 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:19.196495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:19.196501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:19.196661 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:19.197112 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:19.197116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:19.197137 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:19.197637 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:19.197777 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:19.198484 (+ 707us) server_negotiation.cc:299] Negotiation successful
0619 02:27:19.198788 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":117,"threads_started":1}
I20260619 02:27:19.199961 14752 tablet_bootstrap.cc:492] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:27:19.200171 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.184226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:60059)
0619 02:27:19.184738 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:19.184794 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:19.185675 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:19.186733 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:19.186740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:19.187158 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:19.187606 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.187620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.188886 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.188890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.189823 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.189832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.190711 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.190715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.190928 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.191591 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:19.191609 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:19.193470 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:19.196082 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:19.196087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:19.196097 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:19.196346 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:19.196790 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:19.196796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:19.196802 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:19.196988 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:19.197762 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:19.197771 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:19.198104 (+ 333us) client_negotiation.cc:769] Sending connection context
0619 02:27:19.198320 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 02:27:19.198713 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":145,"threads_started":1}
I20260619 02:27:19.200474 14752 tablet_bootstrap.cc:492] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Bootstrap complete.
I20260619 02:27:19.201833 14752 ts_tablet_manager.cc:1397] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.008s
I20260619 02:27:19.202908 14860 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33401
I20260619 02:27:19.203164 14860 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:19.203804 14860 heartbeater.cc:511] Master 127.0.98.62:33401 requested a full tablet report, sending...
I20260619 02:27:19.204658 14752 raft_consensus.cc:348] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:19.205102 14752 raft_consensus.cc:729] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1632bcf8846476c8b09814f67ec7753, State: Initialized, Role: FOLLOWER
I20260619 02:27:19.205634 14752 consensus_queue.cc:260] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:19.206708 14752 ts_tablet_manager.cc:1428] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:27:19.206727 14672 ts_manager.cc:194] Registered new tserver with Master: d1632bcf8846476c8b09814f67ec7753 (127.0.98.1:39357)
I20260619 02:27:19.210266 14672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60059
I20260619 02:27:19.212898 14860 heartbeater.cc:503] Master 127.0.98.62:33401 was elected leader, sending a full tablet report...
Jun 19 02:27:19 dist-test-slave-xkrt krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 02:27:19.220443 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.209618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39357 (local address 127.0.0.1:51228)
0619 02:27:19.210192 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:19.210235 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:19.210381 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:19.210853 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:19.210859 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:19.211326 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:19.211439 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.211445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.212479 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.212486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.213244 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.213255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.214049 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.214066 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.214183 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.214862 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:19.214877 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:19.216591 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:19.218541 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:19.218549 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:19.218555 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:19.218829 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:19.219102 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:19.219105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:19.219107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:19.219198 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:19.219808 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:19.219814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:19.219950 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 02:27:19.220094 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 02:27:19.220268 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":109,"threads_started":1}
I20260619 02:27:19.220757 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.209718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.0.98.1:39357)
0619 02:27:19.210336 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:19.210349 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:19.210443 (+ 94us) server_negotiation.cc:407] Connection header received
0619 02:27:19.210512 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:19.210517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:19.210697 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:19.210825 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:19.211561 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.212364 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.213388 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.213939 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.214078 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.216725 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:19.216733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:19.216743 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:19.216765 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:19.218439 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:19.218936 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:19.218940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:19.218941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:19.219017 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:19.219311 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:19.219316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:19.219317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:19.219667 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:19.219820 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:19.220291 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 02:27:19.220567 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":525,"thread_start_us":104,"threads_started":1}
I20260619 02:27:19.221689 14794 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3ccf3ffb784ba58b2415a22a3d4281 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51228
I20260619 02:27:19.222612 14869 tablet_replica.cc:330] stopping tablet replica
I20260619 02:27:19.222913 14869 raft_consensus.cc:2229] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:27:19.223157 14869 raft_consensus.cc:2258] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:27:19.224184 14869 ts_tablet_manager.cc:1905] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:27:19.228053 14869 ts_tablet_manager.cc:1918] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:27:19.228170 14869 log.cc:1196] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal/wals/2d3ccf3ffb784ba58b2415a22a3d4281
I20260619 02:27:19.228469 14869 ts_tablet_manager.cc:1939] T 2d3ccf3ffb784ba58b2415a22a3d4281 P d1632bcf8846476c8b09814f67ec7753: Deleting consensus metadata
I20260619 02:27:19.230232 14657 catalog_manager.cc:4841] TS d1632bcf8846476c8b09814f67ec7753 (127.0.98.1:39357): tablet 2d3ccf3ffb784ba58b2415a22a3d4281 (table test-table [id=01b5f06f43cf4906ad02c3b144b08774]) successfully deleted
W20260619 02:27:19.309036 14865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:19.309263 14865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:19.309302 14865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:19.313215 14865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:19.313267 14865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:19.313321 14865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:27:19.318210 14865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43645
--tserver_master_addrs=127.0.98.62:33401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:19.319082 14865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:19.320011 14865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:19.329291 14875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:19.329286 14874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:19.329388 14877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:19.329560 14865 server_base.cc:1034] running on GCE node
I20260619 02:27:19.329931 14865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:19.330546 14865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:19.331723 14865 hybrid_clock.cc:648] HybridClock initialized: now 1781836039331714 us; error 29 us; skew 500 ppm
Jun 19 02:27:19 dist-test-slave-xkrt krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:19.334272 14865 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:27:19.335112 14865 webserver.cc:460] Webserver started at http://127.0.98.2:43645/ using document root <none> and password file <none>
I20260619 02:27:19.335639 14865 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:19.335734 14865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:19.339499 14865 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 02:27:19.341522 14884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:19.342725 14865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:27:19.342851 14865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "b15a9351deb148ee9c9a1983b07eb3c4"
format_stamp: "Formatted at 2026-06-19 02:27:18 on dist-test-slave-xkrt"
server_key: "a4a446e4db00dc9bad9e994220c3c6a0"
server_key_iv: "15845a0951b1b5943cdb74fe47d70a37"
server_key_version: "encryptionkey@0"
I20260619 02:27:19.343370 14865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:19.373317 14865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:19.376098 14865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:19.376297 14865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:19.376868 14865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:19.377717 14865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:19.377781 14865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:19.377846 14865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:19.377897 14865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:19.386648 14865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38681
I20260619 02:27:19.386672 14997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38681 every 8 connection(s)
I20260619 02:27:19.388032 14865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:27:19.390518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 14865
I20260619 02:27:19.390844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41505
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33401
--webserver_port=41027
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:27:19 dist-test-slave-xkrt krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:19.402423 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.390022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60743 (local address 127.0.98.62:33401)
0619 02:27:19.390150 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:19.390153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:19.391182 (+ 1029us) server_negotiation.cc:407] Connection header received
0619 02:27:19.392069 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:19.392072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:19.392134 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:19.392212 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:19.393322 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.394251 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.395503 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.395942 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.396073 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.398454 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:19.398460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:19.398465 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:19.398501 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:19.400305 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:19.400823 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:19.400829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:19.400832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:19.400930 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:19.401274 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:19.401278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:19.401280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:19.401558 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:19.401672 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:19.402128 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 02:27:19.402251 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:27:19.403231 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.390336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:60743)
0619 02:27:19.391011 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:19.391060 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:19.391856 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:19.392389 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:19.392396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:19.392723 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:19.393174 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.393187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.394404 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.394408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.395366 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.395376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.396092 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.396094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.396242 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.396843 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:19.396859 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:19.398278 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:19.400430 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:19.400435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:19.400445 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:19.400717 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:19.401037 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:19.401040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:19.401041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:19.401178 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:19.401665 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:19.401670 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:19.401902 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 02:27:19.402063 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 02:27:19.402315 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":116,"threads_started":1}
I20260619 02:27:19.404412 14998 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33401
I20260619 02:27:19.404659 14998 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:19.405180 14998 heartbeater.cc:511] Master 127.0.98.62:33401 requested a full tablet report, sending...
I20260619 02:27:19.406261 14672 ts_manager.cc:194] Registered new tserver with Master: b15a9351deb148ee9c9a1983b07eb3c4 (127.0.98.2:38681)
I20260619 02:27:19.406972 14672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60743
W20260619 02:27:19.498584 15001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:19.498803 15001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:19.498842 15001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:19.502718 15001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:19.502765 15001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:19.502842 15001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:27:19.507921 15001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41027
--tserver_master_addrs=127.0.98.62:33401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:19.508803 15001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:19.509760 15001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:19.518664 15006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:19.518685 15007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:19.518678 15009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:19.518719 15001 server_base.cc:1034] running on GCE node
I20260619 02:27:19.519347 15001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:19.519949 15001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:19.521117 15001 hybrid_clock.cc:648] HybridClock initialized: now 1781836039521080 us; error 51 us; skew 500 ppm
Jun 19 02:27:19 dist-test-slave-xkrt krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:19.523830 15001 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:27:19.524784 15001 webserver.cc:460] Webserver started at http://127.0.98.3:41027/ using document root <none> and password file <none>
I20260619 02:27:19.525324 15001 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:19.525429 15001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:19.529345 15001 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:27:19.531405 15016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:19.532573 15001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:27:19.532702 15001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "f4441ae3c81945b7a028c6903460e4f0"
format_stamp: "Formatted at 2026-06-19 02:27:18 on dist-test-slave-xkrt"
server_key: "b11a40103cbfb6cea1b70345ef6df567"
server_key_iv: "76f5ad34aac31395f35df374ee9e7021"
server_key_version: "encryptionkey@0"
I20260619 02:27:19.533229 15001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:19.553061 15001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:19.556053 15001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:19.556244 15001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:19.556802 15001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:19.557642 15001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:19.557687 15001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:19.557726 15001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:19.557741 15001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:19.566883 15001 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41505
I20260619 02:27:19.566901 15129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41505 every 8 connection(s)
I20260619 02:27:19.567834 15001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:27:19 dist-test-slave-xkrt krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:19.577346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 15001
I20260619 02:27:19.582057 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.569736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42941 (local address 127.0.98.62:33401)
0619 02:27:19.569880 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:19.569885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:19.570907 (+ 1022us) server_negotiation.cc:407] Connection header received
0619 02:27:19.571907 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:19.571911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:19.571970 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:19.572078 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:19.573275 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.573916 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.575109 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.575367 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.575486 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.578074 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:19.578078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:19.578080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:19.578104 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:19.579914 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:19.580502 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:19.580506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:19.580507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:19.580626 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:19.580954 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:19.580958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:19.580960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:19.581180 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:19.581290 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:19.581832 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 02:27:19.581933 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:27:19.583113 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.570110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:42941)
0619 02:27:19.570759 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:19.570799 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:19.571649 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:19.572259 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:19.572266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:19.572627 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:19.573104 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.573117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.574039 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.574043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.574980 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.574989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.575519 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.575523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.575693 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.576221 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:19.576239 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:19.577847 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:19.580024 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:19.580029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:19.580042 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:19.580399 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:19.580731 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:19.580733 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:19.580735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:19.580860 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:19.581298 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:19.581303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:19.581525 (+ 222us) client_negotiation.cc:769] Sending connection context
0619 02:27:19.581770 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 02:27:19.582086 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":113,"threads_started":1}
I20260619 02:27:19.584307 15130 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33401
I20260619 02:27:19.584568 15130 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:19.585059 15130 heartbeater.cc:511] Master 127.0.98.62:33401 requested a full tablet report, sending...
I20260619 02:27:19.586114 14672 ts_manager.cc:194] Registered new tserver with Master: f4441ae3c81945b7a028c6903460e4f0 (127.0.98.3:41505)
I20260619 02:27:19.586652 14672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42941
I20260619 02:27:19.586936 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.578101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.0.98.62:33401)
0619 02:27:19.578347 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:19.578350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:19.578485 (+ 135us) server_negotiation.cc:407] Connection header received
0619 02:27:19.578613 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:19.578617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:19.578671 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:19.578754 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:19.579749 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.580468 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.581699 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.582046 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.582168 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.583771 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:19.583774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:19.583776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:19.583795 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:19.585260 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:19.585721 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:19.585724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:19.585726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:19.585814 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:19.586099 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:19.586102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:19.586104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:19.586325 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:19.586520 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:19.586662 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 02:27:19.586774 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":88,"threads_started":1}
I20260619 02:27:19.588482 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:27:19.592058 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.589033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.0.98.62:33401)
0619 02:27:19.589135 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:19.589138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:19.589319 (+ 181us) server_negotiation.cc:407] Connection header received
0619 02:27:19.589421 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:19.589423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:19.589497 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:19.589628 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:27:19.589778 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.590222 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.590989 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.591188 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.591259 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.591484 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:27:19.591794 (+ 310us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:27:19.591878 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:19.591938 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 02:27:19.591980 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:27:19.592589 14672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:27:19.595784 14672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:27:19.603200 14794 tablet_service.cc:1467] Processing CreateTablet for tablet a0f3672a35cc41bfb11249d6183a4bb2 (DEFAULT_TABLE table=test-table [id=9618b6b180e24f7f8dff3fbd25b7475e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:19.603803 14794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f3672a35cc41bfb11249d6183a4bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:19.606357 14752 tablet_bootstrap.cc:492] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753: Bootstrap starting.
I20260619 02:27:19.607182 14752 tablet_bootstrap.cc:654] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:19.607959 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.602182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41505 (local address 127.0.0.1:56720)
0619 02:27:19.602736 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:19.602766 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:19.602860 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:19.603455 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:19.603460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:19.603479 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:19.603570 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.603577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.605132 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.605136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.606413 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.606423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.607447 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.607451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.607557 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.607605 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 02:27:19.607711 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 02:27:19.607813 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":127,"threads_started":1}
I20260619 02:27:19.607987 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.602371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.0.98.3:41505)
0619 02:27:19.602622 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:19.602632 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:19.602861 (+ 229us) server_negotiation.cc:407] Connection header received
0619 02:27:19.602958 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:19.602963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:19.603136 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:19.603288 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:19.603696 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.605002 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.606518 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.607312 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.607433 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.607606 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:19.607736 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 02:27:19.607840 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20260619 02:27:19.608199 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.601894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38681 (local address 127.0.0.1:47622)
0619 02:27:19.602039 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:19.602053 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:19.602122 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:19.603190 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:19.603193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:19.603211 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:19.603297 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.603303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.604880 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.604884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.606325 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.606337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.607806 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.607809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.607908 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.607930 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:27:19.607985 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 02:27:19.608075 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":31}
I20260619 02:27:19.608793 14752 tablet_bootstrap.cc:492] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753: No bootstrap required, opened a new log
I20260619 02:27:19.608889 14752 ts_tablet_manager.cc:1397] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:27:19.609074 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.602192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.0.98.2:38681)
0619 02:27:19.602639 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:19.602653 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:19.602676 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:27:19.602772 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:19.602778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:19.603020 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:19.603190 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:19.603483 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.604697 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.606798 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.607669 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.608646 (+ 977us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.608792 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:19.608843 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 02:27:19.608932 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":121,"threads_started":1}
I20260619 02:27:19.609333 14752 raft_consensus.cc:348] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:19.609459 14752 raft_consensus.cc:374] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:19.609498 14752 raft_consensus.cc:729] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1632bcf8846476c8b09814f67ec7753, State: Initialized, Role: FOLLOWER
I20260619 02:27:19.609637 14752 consensus_queue.cc:260] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:19.609874 14752 ts_tablet_manager.cc:1428] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:27:19.611462 15064 tablet_service.cc:1467] Processing CreateTablet for tablet a0f3672a35cc41bfb11249d6183a4bb2 (DEFAULT_TABLE table=test-table [id=9618b6b180e24f7f8dff3fbd25b7475e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:19.612103 14932 tablet_service.cc:1467] Processing CreateTablet for tablet a0f3672a35cc41bfb11249d6183a4bb2 (DEFAULT_TABLE table=test-table [id=9618b6b180e24f7f8dff3fbd25b7475e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:19.612463 15064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f3672a35cc41bfb11249d6183a4bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:19.613099 14932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f3672a35cc41bfb11249d6183a4bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:19.618845 15143 tablet_bootstrap.cc:492] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0: Bootstrap starting.
I20260619 02:27:19.619802 15144 tablet_bootstrap.cc:492] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4: Bootstrap starting.
I20260619 02:27:19.621057 15143 tablet_bootstrap.cc:654] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:19.622061 15144 tablet_bootstrap.cc:654] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:19.622474 15143 log.cc:824] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:19.622483 14866 raft_consensus.cc:482] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:27:19.622651 14866 raft_consensus.cc:504] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:19.622871 15144 log.cc:824] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:19.623972 14866 leader_election.cc:288] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b15a9351deb148ee9c9a1983b07eb3c4 (127.0.98.2:38681), f4441ae3c81945b7a028c6903460e4f0 (127.0.98.3:41505)
I20260619 02:27:19.624514 15143 tablet_bootstrap.cc:492] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0: No bootstrap required, opened a new log
I20260619 02:27:19.624699 15143 ts_tablet_manager.cc:1397] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:27:19.625566 15144 tablet_bootstrap.cc:492] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4: No bootstrap required, opened a new log
I20260619 02:27:19.625814 15144 ts_tablet_manager.cc:1397] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260619 02:27:19.628490 15143 raft_consensus.cc:348] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:19.628748 15143 raft_consensus.cc:374] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:19.628811 15143 raft_consensus.cc:729] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4441ae3c81945b7a028c6903460e4f0, State: Initialized, Role: FOLLOWER
I20260619 02:27:19.629321 15143 consensus_queue.cc:260] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:19.629585 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.624136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38681 (local address 127.0.98.1:60017)
0619 02:27:19.624288 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:19.624306 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:19.624405 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:19.625160 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:19.625164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:19.625183 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:19.625291 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.625298 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.626434 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.626438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.627960 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.627971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.629137 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.629141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.629235 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.629282 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 02:27:19.629334 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 02:27:19.629420 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 02:27:19.630227 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.624818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60017 (local address 127.0.98.2:38681)
0619 02:27:19.624941 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:19.624946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:19.624958 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:27:19.625001 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:19.625004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:19.625047 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:19.625132 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:19.625432 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.626288 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.628248 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.629001 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.629983 (+ 982us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.630036 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:19.630070 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 02:27:19.630125 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:27:19.630328 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.624948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56587 (local address 127.0.98.3:41505)
0619 02:27:19.625053 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:19.625057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:19.625071 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:27:19.625119 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:19.625124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:19.625164 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:19.625246 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:19.625702 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.626481 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.628899 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.629607 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.629692 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.629749 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:19.630122 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 02:27:19.630195 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:27:19.630345 15143 ts_tablet_manager.cc:1428] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:27:19.630036 15144 raft_consensus.cc:348] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:19.630671 15144 raft_consensus.cc:374] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:19.630703 15130 heartbeater.cc:503] Master 127.0.98.62:33401 was elected leader, sending a full tablet report...
I20260619 02:27:19.630789 15144 raft_consensus.cc:729] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15a9351deb148ee9c9a1983b07eb3c4, State: Initialized, Role: FOLLOWER
I20260619 02:27:19.630815 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.624409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41505 (local address 127.0.98.1:56587)
0619 02:27:19.624730 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:19.624746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:19.624836 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:19.625349 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:19.625353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:19.625371 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:19.625508 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.625515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.626635 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.626639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.628752 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:19.628763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.629851 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.629854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:19.629964 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.629988 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:27:19.630559 (+ 571us) client_negotiation.cc:240] Negotiation successful
0619 02:27:19.630642 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":3,"thread_start_us":135,"threads_started":1}
I20260619 02:27:19.631083 14952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f3672a35cc41bfb11249d6183a4bb2" candidate_uuid: "d1632bcf8846476c8b09814f67ec7753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" is_pre_election: true
I20260619 02:27:19.631428 15144 consensus_queue.cc:260] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:19.631852 14952 raft_consensus.cc:2454] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1632bcf8846476c8b09814f67ec7753 in term 0.
I20260619 02:27:19.631861 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f3672a35cc41bfb11249d6183a4bb2" candidate_uuid: "d1632bcf8846476c8b09814f67ec7753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4441ae3c81945b7a028c6903460e4f0" is_pre_election: true
I20260619 02:27:19.632122 15084 raft_consensus.cc:2454] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1632bcf8846476c8b09814f67ec7753 in term 0.
I20260619 02:27:19.632428 14748 leader_election.cc:302] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b15a9351deb148ee9c9a1983b07eb3c4, d1632bcf8846476c8b09814f67ec7753; no voters:
I20260619 02:27:19.632504 15144 ts_tablet_manager.cc:1428] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 02:27:19.632877 14998 heartbeater.cc:503] Master 127.0.98.62:33401 was elected leader, sending a full tablet report...
I20260619 02:27:19.632859 14866 raft_consensus.cc:2781] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:27:19.632944 14866 raft_consensus.cc:482] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:27:19.633004 14866 raft_consensus.cc:3037] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:19.634299 14866 raft_consensus.cc:504] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
I20260619 02:27:19.634694 14866 leader_election.cc:288] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [CANDIDATE]: Term 1 election: Requested vote from peers b15a9351deb148ee9c9a1983b07eb3c4 (127.0.98.2:38681), f4441ae3c81945b7a028c6903460e4f0 (127.0.98.3:41505)
I20260619 02:27:19.635262 14952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f3672a35cc41bfb11249d6183a4bb2" candidate_uuid: "d1632bcf8846476c8b09814f67ec7753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15a9351deb148ee9c9a1983b07eb3c4"
I20260619 02:27:19.635385 14952 raft_consensus.cc:3037] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:19.635396 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f3672a35cc41bfb11249d6183a4bb2" candidate_uuid: "d1632bcf8846476c8b09814f67ec7753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4441ae3c81945b7a028c6903460e4f0"
I20260619 02:27:19.635504 15084 raft_consensus.cc:3037] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:19.636685 14952 raft_consensus.cc:2454] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1632bcf8846476c8b09814f67ec7753 in term 1.
I20260619 02:27:19.636852 15084 raft_consensus.cc:2454] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1632bcf8846476c8b09814f67ec7753 in term 1.
I20260619 02:27:19.636981 14748 leader_election.cc:302] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b15a9351deb148ee9c9a1983b07eb3c4, d1632bcf8846476c8b09814f67ec7753; no voters:
I20260619 02:27:19.637172 14866 raft_consensus.cc:2781] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:27:19.637437 14866 raft_consensus.cc:686] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [term 1 LEADER]: Becoming Leader. State: Replica: d1632bcf8846476c8b09814f67ec7753, State: Running, Role: LEADER
I20260619 02:27:19.637712 14866 consensus_queue.cc:237] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } }
W20260619 02:27:19.639132 14999 tablet.cc:2369] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:27:19.640231 14672 catalog_manager.cc:5495] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1632bcf8846476c8b09814f67ec7753 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d1632bcf8846476c8b09814f67ec7753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1632bcf8846476c8b09814f67ec7753" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39357 } health_report { overall_health: HEALTHY } } }
I20260619 02:27:19.651319 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.647834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.0.98.1:39357)
0619 02:27:19.647947 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:19.647951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:19.648002 (+ 51us) server_negotiation.cc:407] Connection header received
0619 02:27:19.648161 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:19.648164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:19.648220 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:19.648347 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:27:19.648571 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.649072 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.649992 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:19.650236 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:19.650339 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:19.650592 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:27:19.650947 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:27:19.651059 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:19.651119 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 02:27:19.651169 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:27:19.654927 14952 raft_consensus.cc:1261] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4 [term 1 FOLLOWER]: Refusing update from remote peer d1632bcf8846476c8b09814f67ec7753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:27:19.654996 15084 raft_consensus.cc:1261] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0 [term 1 FOLLOWER]: Refusing update from remote peer d1632bcf8846476c8b09814f67ec7753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:27:19.655611 15150 consensus_queue.cc:1035] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4441ae3c81945b7a028c6903460e4f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:27:19.655735 14866 consensus_queue.cc:1035] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b15a9351deb148ee9c9a1983b07eb3c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:27:19.665143 15155 mvcc.cc:205] Tried to move back new op lower bound from 7298400418419998720 to 7298400418358165504. Current Snapshot: MvccSnapshot[applied={T|T < 7298400418419998720}]
I20260619 02:27:19.670753 14671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:27:19.670897 14671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:27:19.673067 14671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Sending DeleteTablet for 3 replicas of tablet a0f3672a35cc41bfb11249d6183a4bb2
I20260619 02:27:19.673763 14932 tablet_service.cc:1514] Processing DeleteTablet for tablet a0f3672a35cc41bfb11249d6183a4bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47622
I20260619 02:27:19.673790 15064 tablet_service.cc:1514] Processing DeleteTablet for tablet a0f3672a35cc41bfb11249d6183a4bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56720
I20260619 02:27:19.673890 14794 tablet_service.cc:1514] Processing DeleteTablet for tablet a0f3672a35cc41bfb11249d6183a4bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51228
I20260619 02:27:19.674055 14869 tablet_replica.cc:330] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753: stopping tablet replica
I20260619 02:27:19.674175 14869 raft_consensus.cc:2229] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:27:19.674237 15164 tablet_replica.cc:330] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4: stopping tablet replica
I20260619 02:27:19.674448 15163 tablet_replica.cc:330] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0: stopping tablet replica
I20260619 02:27:19.674475 14869 raft_consensus.cc:2258] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:27:19.674531 15164 raft_consensus.cc:2229] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:27:19.674814 15163 raft_consensus.cc:2229] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:27:19.674837 15164 raft_consensus.cc:2258] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:27:19.675143 15163 raft_consensus.cc:2258] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:27:19.675623 14869 ts_tablet_manager.cc:1905] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:27:19.676295 15164 ts_tablet_manager.cc:1905] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:27:19.676664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 14710
I20260619 02:27:19.679586 15163 ts_tablet_manager.cc:1905] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:27:19.679692 15164 ts_tablet_manager.cc:1918] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:27:19.679812 15164 log.cc:1196] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/a0f3672a35cc41bfb11249d6183a4bb2
I20260619 02:27:19.680168 15164 ts_tablet_manager.cc:1939] T a0f3672a35cc41bfb11249d6183a4bb2 P b15a9351deb148ee9c9a1983b07eb3c4: Deleting consensus metadata
I20260619 02:27:19.681336 14658 catalog_manager.cc:4841] TS b15a9351deb148ee9c9a1983b07eb3c4 (127.0.98.2:38681): tablet a0f3672a35cc41bfb11249d6183a4bb2 (table test-table [id=9618b6b180e24f7f8dff3fbd25b7475e]) successfully deleted
I20260619 02:27:19.681684 15163 ts_tablet_manager.cc:1918] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:27:19.681792 15163 log.cc:1196] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/a0f3672a35cc41bfb11249d6183a4bb2
I20260619 02:27:19.682068 15163 ts_tablet_manager.cc:1939] T a0f3672a35cc41bfb11249d6183a4bb2 P f4441ae3c81945b7a028c6903460e4f0: Deleting consensus metadata
I20260619 02:27:19.682964 14656 catalog_manager.cc:4841] TS f4441ae3c81945b7a028c6903460e4f0 (127.0.98.3:41505): tablet a0f3672a35cc41bfb11249d6183a4bb2 (table test-table [id=9618b6b180e24f7f8dff3fbd25b7475e]) successfully deleted
W20260619 02:27:19.683439 14657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39357 (error 108)
I20260619 02:27:19.684270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 14865
I20260619 02:27:19.684335 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:19.684085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39357 (local address 127.0.0.1:51260)
0619 02:27:19.684196 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:19.684255 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 02:27:19.684504 14657 catalog_manager.cc:4584] TS d1632bcf8846476c8b09814f67ec7753 (127.0.98.1:39357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a0f3672a35cc41bfb11249d6183a4bb2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39357: connect: Connection refused (error 111)
I20260619 02:27:19.690546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 15001
I20260619 02:27:19.695669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 14639
I20260619 02:27:19.701954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33401
--webserver_interface=127.0.98.62
--webserver_port=35563
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:19.809394 15166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:19.809701 15166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:19.809787 15166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:19.814018 15166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:27:19.814116 15166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:19.814157 15166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:19.814195 15166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:27:19.814225 15166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:27:19.819906 15166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:19.820947 15166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:19.821964 15166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:19.830438 15171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:19.830458 15172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:19.830461 15174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:19.830713 15166 server_base.cc:1034] running on GCE node
I20260619 02:27:19.831323 15166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:19.832370 15166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:19.833597 15166 hybrid_clock.cc:648] HybridClock initialized: now 1781836039833587 us; error 36 us; skew 500 ppm
Jun 19 02:27:19 dist-test-slave-xkrt krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:19.836251 15166 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:27:19.837095 15166 webserver.cc:460] Webserver started at http://127.0.98.62:35563/ using document root <none> and password file <none>
I20260619 02:27:19.837656 15166 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:19.837720 15166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:19.841225 15166 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:27:19.843264 15181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:19.844458 15166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 02:27:19.844559 15166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "8af98ff02a1e401a93b1d9a2dd9c2033"
format_stamp: "Formatted at 2026-06-19 02:27:17 on dist-test-slave-xkrt"
server_key: "63bfcdebdbb27d3dd832b7faace22386"
server_key_iv: "deeeab22b52990723e5fd90d2b2154b7"
server_key_version: "encryptionkey@0"
I20260619 02:27:19.845036 15166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:19.871495 15166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:19.874728 15166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:19.874884 15166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:19.882144 15166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33401
I20260619 02:27:19.882167 15233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33401 every 8 connection(s)
I20260619 02:27:19.883138 15166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:27:19.887270 15234 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:27:19.887653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 15166
I20260619 02:27:19.888063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39357
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33401
--webserver_port=41935
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:27:19.888971 15234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Bootstrap starting.
I20260619 02:27:19.906424 15234 log.cc:824] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:19.917466 15234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:27:19.917856 15234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: Bootstrap complete.
I20260619 02:27:19.922255 15234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } }
I20260619 02:27:19.922680 15234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af98ff02a1e401a93b1d9a2dd9c2033, State: Initialized, Role: FOLLOWER
I20260619 02:27:19.923203 15234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } }
I20260619 02:27:19.923352 15234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:27:19.923444 15234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:27:19.923559 15234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 2 FOLLOWER]: Advancing to term 3
I20260619 02:27:19.925414 15234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } }
I20260619 02:27:19.925853 15234 leader_election.cc:302] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8af98ff02a1e401a93b1d9a2dd9c2033; no voters:
I20260619 02:27:19.926159 15234 leader_election.cc:288] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 02:27:19.926326 15238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 02:27:19.926589 15238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [term 3 LEADER]: Becoming Leader. State: Replica: 8af98ff02a1e401a93b1d9a2dd9c2033, State: Running, Role: LEADER
I20260619 02:27:19.926882 15238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } }
I20260619 02:27:19.927350 15234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:27:19.928293 15239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } } }
I20260619 02:27:19.928431 15239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:19.928381 15240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8af98ff02a1e401a93b1d9a2dd9c2033. Latest consensus state: current_term: 3 leader_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af98ff02a1e401a93b1d9a2dd9c2033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33401 } } }
I20260619 02:27:19.928479 15240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:19.932121 15253 catalog_manager.cc:1259] Loaded cluster ID: a8e4dbb50f50422b84dd29e1825ef4e8
I20260619 02:27:19.932188 15253 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: loading cluster ID for follower catalog manager: success
W20260619 02:27:19.932969 15253 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 8af98ff02a1e401a93b1d9a2dd9c2033: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260619 02:27:19.933049 15253 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260619 02:27:19.933364 15247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:27:19.935370 15247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:27:19.935799 15247 catalog_manager.cc:1259] Loaded cluster ID: a8e4dbb50f50422b84dd29e1825ef4e8
I20260619 02:27:19.935870 15247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 02:27:19.936352 15247 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f63dfb379ad google::LogMessage::Fail() at ??:0
@ 0x7f63dfb3a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f63dfb374de google::LogMessage::Flush() at ??:0
@ 0x7f63dfb38a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f63ea476fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f63ea477d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f63ea474579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f63ea4b8640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f63e1d893d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f63e1d89c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f63e1d8b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f63e1d7a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f63e33be6db start_thread at ??:0
@ 0x7f63de63561f clone at ??:0
W20260619 02:27:20.024712 15236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:20.024955 15236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:20.024999 15236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:20.029129 15236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:20.029184 15236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:20.029277 15236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:27:20.034545 15236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41935
--tserver_master_addrs=127.0.98.62:33401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:20.035565 15236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:20.036584 15236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:20.046204 15278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:20.046231 15276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:20.046424 15236 server_base.cc:1034] running on GCE node
W20260619 02:27:20.046258 15275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:20.046818 15236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:20.047456 15236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:20.048635 15236 hybrid_clock.cc:648] HybridClock initialized: now 1781836040048603 us; error 51 us; skew 500 ppm
Jun 19 02:27:20 dist-test-slave-xkrt krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:20.051499 15236 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:27:20.052510 15236 webserver.cc:460] Webserver started at http://127.0.98.1:41935/ using document root <none> and password file <none>
I20260619 02:27:20.053105 15236 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:20.053166 15236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:20.057055 15236 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:27:20.059316 15285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:20.060570 15236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:27:20.060702 15236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "d1632bcf8846476c8b09814f67ec7753"
format_stamp: "Formatted at 2026-06-19 02:27:17 on dist-test-slave-xkrt"
server_key: "f00f681e3f669f9b3bfc38dc9031e3e2"
server_key_iv: "9fe0853058962c3a016295f0b5a4798b"
server_key_version: "encryptionkey@0"
I20260619 02:27:20.061262 15236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:20.079546 15236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:20.082895 15236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:20.083104 15236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:20.083712 15236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:20.084862 15292 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 02:27:20.089628 15292 ts_tablet_manager.cc:1852] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 02:27:20.090164 15292 ts_tablet_manager.cc:1905] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:27:20.095419 15292 ts_tablet_manager.cc:1918] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:27:20.095541 15292 log.cc:1196] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/wal/wals/a0f3672a35cc41bfb11249d6183a4bb2
I20260619 02:27:20.095932 15292 ts_tablet_manager.cc:1939] T a0f3672a35cc41bfb11249d6183a4bb2 P d1632bcf8846476c8b09814f67ec7753: Deleting consensus metadata
I20260619 02:27:20.096256 15236 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 02:27:20.096344 15236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.000s sys 0.000s
I20260619 02:27:20.096470 15236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:20.096518 15236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:20.107836 15236 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39357
I20260619 02:27:20.107869 15399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39357 every 8 connection(s)
I20260619 02:27:20.108765 15236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:27:20.114655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 15236
I20260619 02:27:20.115118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38681
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33401
--webserver_port=43645
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:20.142700 15402 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:27:20.111097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:51483)
0619 02:27:20.111609 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:20.111660 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:20.112491 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:20.141739 (+ 29248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":140,"threads_started":1}
W20260619 02:27:20.143153 15400 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33401 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33401: Client connection negotiation failed: client connection to 127.0.98.62:33401: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:27:20.143898 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:20.143624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:58129)
0619 02:27:20.143762 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:20.143835 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 02:27:20.144611 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:20.144372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:60831)
0619 02:27:20.144516 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:20.144563 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260619 02:27:20.144909 15400 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:27:20.224447 15403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:20.224725 15403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:20.224799 15403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:20.228874 15403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:20.228971 15403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:20.229058 15403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:27:20.234337 15403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43645
--tserver_master_addrs=127.0.98.62:33401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:20.235376 15403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:20.236370 15403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:20.248222 15411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:20.248308 15403 server_base.cc:1034] running on GCE node
W20260619 02:27:20.248216 15409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:20.248551 15408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:20.248980 15403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:20.249675 15403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:20.250852 15403 hybrid_clock.cc:648] HybridClock initialized: now 1781836040250809 us; error 64 us; skew 500 ppm
Jun 19 02:27:20 dist-test-slave-xkrt krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:20.253463 15403 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:27:20.254369 15403 webserver.cc:460] Webserver started at http://127.0.98.2:43645/ using document root <none> and password file <none>
I20260619 02:27:20.254938 15403 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:20.255030 15403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:20.258693 15403 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:27:20.260610 15418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:20.261849 15403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:27:20.261982 15403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "b15a9351deb148ee9c9a1983b07eb3c4"
format_stamp: "Formatted at 2026-06-19 02:27:18 on dist-test-slave-xkrt"
server_key: "a4a446e4db00dc9bad9e994220c3c6a0"
server_key_iv: "15845a0951b1b5943cdb74fe47d70a37"
server_key_version: "encryptionkey@0"
I20260619 02:27:20.262519 15403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:20.293392 15403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:20.296275 15403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:20.296478 15403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:20.297066 15403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:20.297874 15403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:20.297955 15403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:20.298019 15403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:20.298062 15403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:20.306809 15403 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38681
I20260619 02:27:20.306838 15531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38681 every 8 connection(s)
I20260619 02:27:20.307762 15403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:27:20.311170 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:20.309801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:33429)
0619 02:27:20.310425 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:20.310527 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":107,"threads_started":1}
I20260619 02:27:20.311606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 15403
W20260619 02:27:20.311631 15532 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33401 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33401: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
I20260619 02:27:20.312069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41505
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33401
--webserver_port=41027
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:27:20.312253 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:20.311998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:41485)
0619 02:27:20.312107 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:20.312173 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 02:27:20.312856 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:20.312659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:59391)
0619 02:27:20.312763 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:20.312812 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 02:27:20.313042 15532 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:27:20.421540 15535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:20.421814 15535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:20.421854 15535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:20.425736 15535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:20.425781 15535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:20.425868 15535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:27:20.430984 15535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41027
--tserver_master_addrs=127.0.98.62:33401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:20.431886 15535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:20.432832 15535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:20.441862 15543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:20.441857 15541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:20.442044 15535 server_base.cc:1034] running on GCE node
W20260619 02:27:20.441952 15540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:20.442560 15535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:20.443164 15535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:20.444350 15535 hybrid_clock.cc:648] HybridClock initialized: now 1781836040444325 us; error 41 us; skew 500 ppm
Jun 19 02:27:20 dist-test-slave-xkrt krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:20.446980 15535 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:27:20.447844 15535 webserver.cc:460] Webserver started at http://127.0.98.3:41027/ using document root <none> and password file <none>
I20260619 02:27:20.448398 15535 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:20.448486 15535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:20.452049 15535 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:27:20.453933 15550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:20.455039 15535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:27:20.455170 15535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "f4441ae3c81945b7a028c6903460e4f0"
format_stamp: "Formatted at 2026-06-19 02:27:18 on dist-test-slave-xkrt"
server_key: "b11a40103cbfb6cea1b70345ef6df567"
server_key_iv: "76f5ad34aac31395f35df374ee9e7021"
server_key_version: "encryptionkey@0"
I20260619 02:27:20.455657 15535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:20.486008 15535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:20.489073 15535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:20.489271 15535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:20.489852 15535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:20.490651 15535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:20.490718 15535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:20.490788 15535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:20.490841 15535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:20.499672 15535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41505
I20260619 02:27:20.499697 15663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41505 every 8 connection(s)
I20260619 02:27:20.500633 15535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:27:20.504307 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:20.502738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:56023)
0619 02:27:20.503412 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:20.503521 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":106,"threads_started":1}
W20260619 02:27:20.504873 15664 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33401 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33401: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
I20260619 02:27:20.505434 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:20.505192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:33203)
0619 02:27:20.505318 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:20.505372 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 02:27:20.506059 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:20.505858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:54947)
0619 02:27:20.505959 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:20.506005 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 02:27:20.506223 15664 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:27:20.508313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 15535
I20260619 02:27:21.145881 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:21.145408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:35533)
0619 02:27:21.145721 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:21.145778 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":69,"threads_started":1}
I20260619 02:27:21.313908 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:21.313500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:57639)
0619 02:27:21.313770 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:21.313819 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":129,"threads_started":1}
I20260619 02:27:21.507102 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:21.506720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:44535)
0619 02:27:21.506961 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:21.507011 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
I20260619 02:27:22.147075 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:22.146567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:58985)
0619 02:27:22.146871 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:22.146958 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":108,"threads_started":1}
I20260619 02:27:22.314998 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:22.314604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:47701)
0619 02:27:22.314850 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:22.314915 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
I20260619 02:27:22.508237 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:22.507851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:32827)
0619 02:27:22.508103 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:22.508155 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260619 02:27:23.148240 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:23.147831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:51955)
0619 02:27:23.148075 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:23.148128 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":82,"threads_started":1}
I20260619 02:27:23.316210 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:23.315800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:53469)
0619 02:27:23.316068 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:23.316121 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":67,"threads_started":1}
I20260619 02:27:23.509322 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:23.508923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:47673)
0619 02:27:23.509164 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:23.509220 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260619 02:27:24.149403 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:24.149006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:39843)
0619 02:27:24.149250 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:24.149309 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260619 02:27:24.317269 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:24.316885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:36815)
0619 02:27:24.317119 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:24.317172 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260619 02:27:24.510408 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:24.510038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:43163)
0619 02:27:24.510264 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:24.510312 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":73,"threads_started":1}
I20260619 02:27:25.150671 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:25.150224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:44541)
0619 02:27:25.150503 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:25.150564 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":115,"threads_started":1}
I20260619 02:27:25.318415 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:25.318029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:55785)
0619 02:27:25.318271 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:25.318325 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260619 02:27:25.511543 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:25.511142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:36569)
0619 02:27:25.511404 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:25.511456 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
I20260619 02:27:26.151988 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:26.151521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:45495)
0619 02:27:26.151820 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:26.151884 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
I20260619 02:27:26.319509 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:26.319101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:56205)
0619 02:27:26.319363 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:26.319418 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":85,"threads_started":1}
I20260619 02:27:26.512698 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:26.512327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:51437)
0619 02:27:26.512557 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:26.512610 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260619 02:27:27.153147 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:27.152696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:44179)
0619 02:27:27.152999 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:27.153051 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":72,"threads_started":1}
I20260619 02:27:27.320711 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:27.320247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:55541)
0619 02:27:27.320570 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:27.320622 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":102,"threads_started":1}
I20260619 02:27:27.513815 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:27.513372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:42871)
0619 02:27:27.513678 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:27.513728 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":68,"threads_started":1}
I20260619 02:27:28.154484 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:28.154043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:60759)
0619 02:27:28.154325 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:28.154385 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260619 02:27:28.321768 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:28.321369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:45143)
0619 02:27:28.321630 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:28.321681 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":70,"threads_started":1}
I20260619 02:27:28.514950 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:28.514534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:37407)
0619 02:27:28.514783 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:28.514837 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260619 02:27:29.155781 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:29.155306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:51993)
0619 02:27:29.155608 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:29.155666 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":101,"threads_started":1}
I20260619 02:27:29.322914 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:29.322520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:47463)
0619 02:27:29.322767 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:29.322818 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":88,"threads_started":1}
I20260619 02:27:29.516158 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:29.515755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:40331)
0619 02:27:29.516003 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:29.516063 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
I20260619 02:27:30.157023 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:30.156591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:37727)
0619 02:27:30.156872 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:30.156926 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":89,"threads_started":1}
I20260619 02:27:30.323992 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:30.323614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:44207)
0619 02:27:30.323850 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:30.323907 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
I20260619 02:27:30.517365 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:30.516928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:56539)
0619 02:27:30.517216 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:30.517270 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":119,"threads_started":1}
I20260619 02:27:31.158259 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:31.157816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:45845)
0619 02:27:31.158102 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:31.158161 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
I20260619 02:27:31.325136 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:31.324693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:58355)
0619 02:27:31.324991 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:31.325044 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":134,"threads_started":1}
I20260619 02:27:31.518718 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:31.518254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:45291)
0619 02:27:31.518556 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:31.518622 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
I20260619 02:27:32.159523 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:32.159017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:56077)
0619 02:27:32.159306 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:32.159363 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":89,"threads_started":1}
I20260619 02:27:32.326284 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:32.325896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:33231)
0619 02:27:32.326145 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:32.326194 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20260619 02:27:32.519829 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:32.519423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:45783)
0619 02:27:32.519686 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:32.519736 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":98,"threads_started":1}
I20260619 02:27:33.160686 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:33.160257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:38385)
0619 02:27:33.160528 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:33.160590 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":90,"threads_started":1}
I20260619 02:27:33.327396 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:33.327016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:32925)
0619 02:27:33.327255 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:33.327308 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260619 02:27:33.520972 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:33.520525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:41465)
0619 02:27:33.520782 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:33.520877 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":105,"threads_started":1}
I20260619 02:27:34.161864 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:34.161422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:40419)
0619 02:27:34.161713 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:34.161769 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":74,"threads_started":1}
I20260619 02:27:34.328532 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:34.328121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:46033)
0619 02:27:34.328378 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:34.328439 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260619 02:27:34.522176 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:34.521754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:42279)
0619 02:27:34.522001 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:34.522069 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":69,"threads_started":1}
I20260619 02:27:35.163084 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:35.162687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.1:34557)
0619 02:27:35.162934 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:35.162985 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260619 02:27:35.329854 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:35.329398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.2:33303)
0619 02:27:35.329675 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:35.329727 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":70,"threads_started":1}
I20260619 02:27:35.513726 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 15236
I20260619 02:27:35.519174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 15403
I20260619 02:27:35.523530 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:35.523028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33401 (local address 127.0.98.3:44275)
0619 02:27:35.523314 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:35.523391 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":63,"threads_started":1}
I20260619 02:27:35.524681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 15535
2026-06-19T02:27:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19240 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:27:35 dist-test-slave-xkrt krb5kdc[15715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:27:35 dist-test-slave-xkrt krb5kdc[15715](info): set up 2 sockets
Jun 19 02:27:35 dist-test-slave-xkrt krb5kdc[15715](info): commencing operation
krb5kdc: starting...
W20260619 02:27:36.577862 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:27:36 dist-test-slave-xkrt krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:27:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:27:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:27:36.696295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40637
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:36.803776 15728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:36.804020 15728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:36.804059 15728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:36.807982 15728 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:27:36.808030 15728 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:27:36.808046 15728 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:27:36.808059 15728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:27:36.808076 15728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:36.808125 15728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:36.808161 15728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:27:36.808180 15728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:27:36.813234 15728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:36.814253 15728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:36.815249 15728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:36.823923 15733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:36.823966 15728 server_base.cc:1034] running on GCE node
W20260619 02:27:36.823890 15734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:36.823905 15736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:36.824678 15728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:36.825829 15728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:36.827020 15728 hybrid_clock.cc:648] HybridClock initialized: now 1781836056827008 us; error 26 us; skew 500 ppm
Jun 19 02:27:36 dist-test-slave-xkrt krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:36.829705 15728 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:27:36.830529 15728 webserver.cc:460] Webserver started at http://127.0.98.62:33015/ using document root <none> and password file <none>
I20260619 02:27:36.831050 15728 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:36.831105 15728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:36.831259 15728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:27:36.833151 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "247d63f901dd4743b8114523694b4402"
format_stamp: "Formatted at 2026-06-19 02:27:36 on dist-test-slave-xkrt"
server_key: "8b76c1e825e2daee14e34086e729d59d"
server_key_iv: "2fafd586f58ca0f64eee3d4b524f8d47"
server_key_version: "encryptionkey@0"
I20260619 02:27:36.833702 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "247d63f901dd4743b8114523694b4402"
format_stamp: "Formatted at 2026-06-19 02:27:36 on dist-test-slave-xkrt"
server_key: "8b76c1e825e2daee14e34086e729d59d"
server_key_iv: "2fafd586f58ca0f64eee3d4b524f8d47"
server_key_version: "encryptionkey@0"
I20260619 02:27:36.837205 15728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:27:36.839497 15743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:36.840727 15728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:27:36.840890 15728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "247d63f901dd4743b8114523694b4402"
format_stamp: "Formatted at 2026-06-19 02:27:36 on dist-test-slave-xkrt"
server_key: "8b76c1e825e2daee14e34086e729d59d"
server_key_iv: "2fafd586f58ca0f64eee3d4b524f8d47"
server_key_version: "encryptionkey@0"
I20260619 02:27:36.840977 15728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:36.856740 15728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:36.859529 15728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:36.859674 15728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:36.866357 15728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40637
I20260619 02:27:36.866636 15795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40637 every 8 connection(s)
I20260619 02:27:36.867318 15728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:27:36.870042 15796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:36.871788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 15728
I20260619 02:27:36.871912 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:27:36.872215 392 external_mini_cluster.cc:1428] Setting key a15cebc20fc8f0c43ec96aaccd03ffb7
I20260619 02:27:36.876139 15796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Bootstrap starting.
I20260619 02:27:36.878242 15796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:36.878914 15796 log.cc:824] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:36.880725 15796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: No bootstrap required, opened a new log
Jun 19 02:27:36 dist-test-slave-xkrt krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:36.885090 15796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } }
I20260619 02:27:36.885340 15796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:36.885429 15796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247d63f901dd4743b8114523694b4402, State: Initialized, Role: FOLLOWER
I20260619 02:27:36.885908 15796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } }
I20260619 02:27:36.886058 15796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:27:36.886127 15796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:27:36.886221 15796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:36.886231 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:36.873626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46290 (local address 127.0.98.62:40637)
0619 02:27:36.874038 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:36.874051 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:36.874086 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:27:36.874759 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:36.874776 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:36.875078 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:36.875422 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:36.876253 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:36.877234 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:36.878174 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:36.878695 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:36.878869 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:36.881502 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:36.881507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:36.881521 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:36.881595 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:36.883265 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:36.883759 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:36.883774 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:36.883778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:36.883880 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:36.884159 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:36.884161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:36.884162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:36.884520 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:36.884668 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:36.885009 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 02:27:36.885376 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":128,"threads_started":1}
I20260619 02:27:36.887092 15796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } }
I20260619 02:27:36.887435 15796 leader_election.cc:302] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 247d63f901dd4743b8114523694b4402; no voters:
I20260619 02:27:36.887751 15796 leader_election.cc:288] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:27:36.887825 15801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:27:36.888025 15801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 1 LEADER]: Becoming Leader. State: Replica: 247d63f901dd4743b8114523694b4402, State: Running, Role: LEADER
I20260619 02:27:36.888350 15801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } }
I20260619 02:27:36.888947 15796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:27:36.889897 15803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 247d63f901dd4743b8114523694b4402. Latest consensus state: current_term: 1 leader_uuid: "247d63f901dd4743b8114523694b4402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } } }
I20260619 02:27:36.890009 15803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:36.890398 15810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:27:36.890447 15802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "247d63f901dd4743b8114523694b4402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } } }
I20260619 02:27:36.890602 15802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:36.893703 15810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:27:36.899365 15810 catalog_manager.cc:1347] Generated new cluster ID: 2b71230bf7b644db8db928f16fc04460
I20260619 02:27:36.899422 15810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:27:36.909716 15810 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:27:36.910609 15810 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:27:36.917030 15810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Generated new TSK 0
I20260619 02:27:36.917837 15810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:27:36.951980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40637
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:37.059623 15822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:37.059901 15822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:37.059979 15822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:37.063948 15822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:37.064036 15822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:37.064126 15822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:27:37.069406 15822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:37.070518 15822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:37.071524 15822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:37.081144 15827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:37.081161 15830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:37.081317 15828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:37.081337 15822 server_base.cc:1034] running on GCE node
I20260619 02:27:37.081925 15822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:37.082602 15822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:37.083830 15822 hybrid_clock.cc:648] HybridClock initialized: now 1781836057083803 us; error 38 us; skew 500 ppm
Jun 19 02:27:37 dist-test-slave-xkrt krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:37.086745 15822 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:27:37.087632 15822 webserver.cc:460] Webserver started at http://127.0.98.1:46513/ using document root <none> and password file <none>
I20260619 02:27:37.088155 15822 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:37.088213 15822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:37.088387 15822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:27:37.090523 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "58b66e078b9f44628e792344f37eb35a"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "88854bd6dc221055c449acfef2691bb4"
server_key_iv: "ec257d079309e599c1a0fd08648d531a"
server_key_version: "encryptionkey@0"
I20260619 02:27:37.091027 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "58b66e078b9f44628e792344f37eb35a"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "88854bd6dc221055c449acfef2691bb4"
server_key_iv: "ec257d079309e599c1a0fd08648d531a"
server_key_version: "encryptionkey@0"
I20260619 02:27:37.094589 15822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:27:37.096829 15837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:37.098172 15822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:27:37.098269 15822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "58b66e078b9f44628e792344f37eb35a"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "88854bd6dc221055c449acfef2691bb4"
server_key_iv: "ec257d079309e599c1a0fd08648d531a"
server_key_version: "encryptionkey@0"
I20260619 02:27:37.098378 15822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:37.115392 15822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:37.118253 15822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:37.118412 15822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:37.118901 15822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:37.119660 15822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:37.119706 15822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:37.119745 15822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:37.119760 15822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:37.128331 15822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36225
I20260619 02:27:37.128361 15950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36225 every 8 connection(s)
I20260619 02:27:37.129159 15822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:27:37.137946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 15822
I20260619 02:27:37.138078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:27:37.138372 392 external_mini_cluster.cc:1428] Setting key a2af61fcf6083a7fee6386d4d843319e
Jun 19 02:27:37 dist-test-slave-xkrt krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:37.142604 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.130859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45321 (local address 127.0.98.62:40637)
0619 02:27:37.130987 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:37.130997 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:37.131727 (+ 730us) server_negotiation.cc:407] Connection header received
0619 02:27:37.133039 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:37.133042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:37.133101 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:37.133204 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:37.134469 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.135058 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.136247 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.136472 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.136547 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.139074 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:37.139082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:37.139084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:37.139127 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:37.140597 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:37.141105 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:37.141109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:37.141111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:37.141190 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:37.141525 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:37.141528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:37.141530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:37.141735 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:37.141843 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:37.142300 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 02:27:37.142434 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:27:37.143330 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.131173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:45321)
0619 02:27:37.131583 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:37.131624 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:37.132832 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:37.133369 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:37.133375 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:37.133831 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:37.134287 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.134300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.135191 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.135194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.136140 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.136149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.136592 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.136594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.136725 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.137253 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:37.137268 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:37.138908 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:37.140733 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:37.140738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:37.140751 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:37.141004 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:37.141288 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:37.141291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:37.141293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:37.141426 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:37.141842 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:37.141848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:37.142080 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 02:27:37.142259 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 02:27:37.142554 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":112,"threads_started":1}
I20260619 02:27:37.144476 15951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40637
I20260619 02:27:37.144708 15951 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:37.145167 15951 heartbeater.cc:511] Master 127.0.98.62:40637 requested a full tablet report, sending...
I20260619 02:27:37.146641 15760 ts_manager.cc:194] Registered new tserver with Master: 58b66e078b9f44628e792344f37eb35a (127.0.98.1:36225)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:27:37.147801 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:27:37.164443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40637
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:37.274245 15956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:37.274480 15956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:37.274520 15956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:37.278571 15956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:37.278623 15956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:37.278707 15956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:27:37.283855 15956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:37.284801 15956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:37.285800 15956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:37.295102 15962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:37.295116 15961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:37.295305 15956 server_base.cc:1034] running on GCE node
W20260619 02:27:37.295178 15964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:37.295760 15956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:37.296327 15956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:37.297484 15956 hybrid_clock.cc:648] HybridClock initialized: now 1781836057297467 us; error 29 us; skew 500 ppm
Jun 19 02:27:37 dist-test-slave-xkrt krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:37.300045 15956 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:27:37.300763 15956 webserver.cc:460] Webserver started at http://127.0.98.2:42995/ using document root <none> and password file <none>
I20260619 02:27:37.301241 15956 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:37.301303 15956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:37.301450 15956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:27:37.303390 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "74ab934f359c4ed7be3cba8cfef0056e"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "fbad55d1108b6dea1439fb14bba27c67"
server_key_iv: "f95ac784d50c94ca127d0214dfa062f9"
server_key_version: "encryptionkey@0"
I20260619 02:27:37.303884 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "74ab934f359c4ed7be3cba8cfef0056e"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "fbad55d1108b6dea1439fb14bba27c67"
server_key_iv: "f95ac784d50c94ca127d0214dfa062f9"
server_key_version: "encryptionkey@0"
I20260619 02:27:37.307243 15956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:27:37.309357 15971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:37.310539 15956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:27:37.310667 15956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "74ab934f359c4ed7be3cba8cfef0056e"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "fbad55d1108b6dea1439fb14bba27c67"
server_key_iv: "f95ac784d50c94ca127d0214dfa062f9"
server_key_version: "encryptionkey@0"
I20260619 02:27:37.310771 15956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:37.332755 15956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:37.335430 15956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:37.335626 15956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:37.336184 15956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:37.336969 15956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:37.337040 15956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:37.337112 15956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:37.337155 15956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:37.346279 15956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44497
I20260619 02:27:37.346307 16084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44497 every 8 connection(s)
I20260619 02:27:37.347136 15956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:27:37.350172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 15956
I20260619 02:27:37.350265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:27:37.350589 392 external_mini_cluster.cc:1428] Setting key d1877ffb3aa147c03e13d13e9188564d
Jun 19 02:27:37 dist-test-slave-xkrt krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:27:37.361266 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.348823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58873 (local address 127.0.98.62:40637)
0619 02:27:37.348980 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:37.348984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:37.349773 (+ 789us) server_negotiation.cc:407] Connection header received
0619 02:27:37.351250 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:37.351253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:37.351308 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:37.351414 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:37.352679 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.353200 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.354391 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.354800 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.354922 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.357469 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:37.357475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:37.357479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:37.357512 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:37.359223 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:37.359722 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:37.359728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:37.359731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:37.359815 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:37.360195 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:37.360198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:37.360199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:37.360379 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:37.360490 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:37.360981 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 02:27:37.361100 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:27:37.362125 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.349176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:58873)
0619 02:27:37.349638 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:37.349678 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:37.351030 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:37.351578 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:37.351585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:37.352080 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:37.352542 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.352555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.353344 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.353346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.354269 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.354278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.354971 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.354974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.355129 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.355807 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:37.355832 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:37.357299 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:37.359357 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:37.359362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:37.359374 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:37.359620 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:37.359921 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:37.359923 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:37.359925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:37.360052 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:37.360487 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:37.360493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:37.360721 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 02:27:37.360943 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 02:27:37.361239 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 02:27:37.363289 16085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40637
I20260619 02:27:37.363528 16085 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:37.364127 16085 heartbeater.cc:511] Master 127.0.98.62:40637 requested a full tablet report, sending...
I20260619 02:27:37.365195 15760 ts_manager.cc:194] Registered new tserver with Master: 74ab934f359c4ed7be3cba8cfef0056e (127.0.98.2:44497)
I20260619 02:27:37.365819 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:27:37.376947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40637
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:37.486990 16090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:37.487210 16090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:37.487274 16090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:37.491508 16090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:37.491575 16090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:37.491675 16090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:27:37.496682 16090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:37.497732 16090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:37.498653 16090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:37.508031 16095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:37.508108 16090 server_base.cc:1034] running on GCE node
W20260619 02:27:37.508031 16096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:37.508246 16098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:37.508590 16090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:37.509200 16090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:37.510382 16090 hybrid_clock.cc:648] HybridClock initialized: now 1781836057510360 us; error 38 us; skew 500 ppm
Jun 19 02:27:37 dist-test-slave-xkrt krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:37.512953 16090 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:27:37.513849 16090 webserver.cc:460] Webserver started at http://127.0.98.3:37489/ using document root <none> and password file <none>
I20260619 02:27:37.514369 16090 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:37.514470 16090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:37.514693 16090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:27:37.516570 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "c8fc8a1ef352471b9d3359580371ea24"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "4b7c920e8b674f0bce387abe35eee731"
server_key_iv: "f13518fddf8e465c6d7336f4b7a7c331"
server_key_version: "encryptionkey@0"
I20260619 02:27:37.517113 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c8fc8a1ef352471b9d3359580371ea24"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "4b7c920e8b674f0bce387abe35eee731"
server_key_iv: "f13518fddf8e465c6d7336f4b7a7c331"
server_key_version: "encryptionkey@0"
I20260619 02:27:37.520548 16090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:27:37.522832 16105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:37.523998 16090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:27:37.524123 16090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "c8fc8a1ef352471b9d3359580371ea24"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "4b7c920e8b674f0bce387abe35eee731"
server_key_iv: "f13518fddf8e465c6d7336f4b7a7c331"
server_key_version: "encryptionkey@0"
I20260619 02:27:37.524226 16090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:37.544080 16090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:37.547037 16090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:37.547237 16090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:37.547838 16090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:37.548638 16090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:37.548707 16090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:37.548779 16090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:37.548820 16090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:37.557904 16090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38555
I20260619 02:27:37.557941 16218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38555 every 8 connection(s)
I20260619 02:27:37.558775 16090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:27:37.562443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 16090
I20260619 02:27:37.562579 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:27:37.562840 392 external_mini_cluster.cc:1428] Setting key 6156b824a14d6521e41250941fc4cd1b
Jun 19 02:27:37 dist-test-slave-xkrt krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:37.572295 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.560512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52259 (local address 127.0.98.62:40637)
0619 02:27:37.560651 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:37.560655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:37.561838 (+ 1183us) server_negotiation.cc:407] Connection header received
0619 02:27:37.562771 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:37.562774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:37.562841 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:37.562945 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:37.564477 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.565142 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.566401 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.566614 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.566718 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.569104 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:37.569107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:37.569109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:37.569158 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:37.570581 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:37.571047 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:37.571049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:37.571050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:37.571120 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:37.571434 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:37.571437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:37.571438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:37.571607 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:37.571690 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:37.572097 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 02:27:37.572186 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:27:37.573072 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.560790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:52259)
0619 02:27:37.561680 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:37.561721 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:37.562521 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:37.563141 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:37.563150 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:37.563657 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:37.564339 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.564359 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.565252 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.565255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.566219 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.566228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.566759 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.566762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.566911 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.567539 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:37.567554 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:37.568949 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:37.570688 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:37.570693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:37.570703 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:37.570959 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:37.571198 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:37.571200 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:37.571201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:37.571330 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:37.571699 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:37.571703 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:37.571906 (+ 203us) client_negotiation.cc:769] Sending connection context
0619 02:27:37.572052 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 02:27:37.572290 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":531,"threads_started":1}
I20260619 02:27:37.574173 16219 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40637
I20260619 02:27:37.574437 16219 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:37.574891 16219 heartbeater.cc:511] Master 127.0.98.62:40637 requested a full tablet report, sending...
I20260619 02:27:37.576036 15760 ts_manager.cc:194] Registered new tserver with Master: c8fc8a1ef352471b9d3359580371ea24 (127.0.98.3:38555)
I20260619 02:27:37.576478 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:27:37.576720 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52259
I20260619 02:27:37.585745 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.578542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46296 (local address 127.0.98.62:40637)
0619 02:27:37.578680 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:37.578686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:37.578829 (+ 143us) server_negotiation.cc:407] Connection header received
0619 02:27:37.578963 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:37.578966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:37.579004 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:37.579096 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:37.579744 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.580231 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.581094 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.581375 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.581483 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.582840 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:37.582843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:37.582847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:37.582870 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:37.584076 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:37.584500 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:37.584504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:37.584506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:37.584588 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:37.584879 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:37.584884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:37.584885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:37.585092 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:37.585237 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:37.585462 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 02:27:37.585596 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:27:37.588296 15760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:27:37.591612 15760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:27:37.605888 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.599723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36225 (local address 127.0.0.1:58650)
0619 02:27:37.600841 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:37.600855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:37.600931 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:37.601854 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:37.601857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:37.601876 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:37.601983 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.601989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.603471 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.603475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.604582 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.604589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.605427 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.605430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.605561 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.605619 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 02:27:37.605694 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:27:37.605755 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"mutex_wait_us":84,"thread_start_us":95,"threads_started":1}
I20260619 02:27:37.605875 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.600552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.0.98.3:38555)
0619 02:27:37.601164 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:37.601178 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:37.601201 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:27:37.601272 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:37.601276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:37.601460 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:37.601665 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:37.601848 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.603039 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.604213 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.605014 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.605126 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.605280 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:37.605623 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 02:27:37.605727 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":156,"threads_started":1}
I20260619 02:27:37.605907 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.600413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38555 (local address 127.0.0.1:59522)
0619 02:27:37.601068 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:37.601079 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:37.601150 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:37.601637 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:37.601640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:37.601658 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:37.601749 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.601755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.603179 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.603182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.604092 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.604101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.605139 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.605143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.605249 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.605291 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 02:27:37.605407 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 02:27:37.605518 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260619 02:27:37.606458 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.600712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.0.98.1:36225)
0619 02:27:37.601380 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:37.601394 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:37.601414 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:27:37.601484 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:37.601489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:37.601705 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:37.602147 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:37.602286 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.603342 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.604696 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.605326 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.605438 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.605589 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:37.605732 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 02:27:37.605826 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":80,"threads_started":1}
I20260619 02:27:37.606719 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.599555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44497 (local address 127.0.0.1:34118)
0619 02:27:37.600234 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:37.600294 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:37.600569 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:37.601127 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:37.601133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:37.601156 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:37.601294 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.601301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.602550 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.602553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.604241 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.604251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.605393 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.605399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.606441 (+ 1042us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.606469 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 02:27:37.606538 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 02:27:37.606602 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":932,"thread_start_us":74,"threads_started":1}
I20260619 02:27:37.607106 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.599718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.0.98.2:44497)
0619 02:27:37.600263 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:37.600275 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:37.600366 (+ 91us) server_negotiation.cc:407] Connection header received
0619 02:27:37.600769 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:37.600776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:37.600971 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:37.601127 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:37.601419 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.602412 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.604360 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.605120 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.605208 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.605313 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:37.606870 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0619 02:27:37.606957 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":180,"threads_started":1}
I20260619 02:27:37.609198 15885 tablet_service.cc:1467] Processing CreateTablet for tablet 75553a82024e42189bb2591516a33a9e (DEFAULT_TABLE table=test-table [id=95b68e0a6ca14f64bf50cd135cf75829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:37.609198 16153 tablet_service.cc:1467] Processing CreateTablet for tablet 75553a82024e42189bb2591516a33a9e (DEFAULT_TABLE table=test-table [id=95b68e0a6ca14f64bf50cd135cf75829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:37.609400 16019 tablet_service.cc:1467] Processing CreateTablet for tablet 75553a82024e42189bb2591516a33a9e (DEFAULT_TABLE table=test-table [id=95b68e0a6ca14f64bf50cd135cf75829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:37.610303 16153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75553a82024e42189bb2591516a33a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:37.610299 15885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75553a82024e42189bb2591516a33a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:37.610440 16019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75553a82024e42189bb2591516a33a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:37.616703 16236 tablet_bootstrap.cc:492] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e: Bootstrap starting.
I20260619 02:27:37.617462 16238 tablet_bootstrap.cc:492] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Bootstrap starting.
I20260619 02:27:37.617796 16237 tablet_bootstrap.cc:492] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24: Bootstrap starting.
I20260619 02:27:37.619467 16237 tablet_bootstrap.cc:654] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:37.619572 16238 tablet_bootstrap.cc:654] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:37.620070 16236 tablet_bootstrap.cc:654] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:37.620733 16237 log.cc:824] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:37.620800 16238 log.cc:824] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:37.620898 16236 log.cc:824] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:37.622805 16238 tablet_bootstrap.cc:492] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: No bootstrap required, opened a new log
I20260619 02:27:37.622821 16237 tablet_bootstrap.cc:492] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24: No bootstrap required, opened a new log
I20260619 02:27:37.622995 16237 ts_tablet_manager.cc:1397] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 02:27:37.622996 16238 ts_tablet_manager.cc:1397] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:27:37.626587 16236 tablet_bootstrap.cc:492] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e: No bootstrap required, opened a new log
I20260619 02:27:37.626824 16236 ts_tablet_manager.cc:1397] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 02:27:37.626878 16237 raft_consensus.cc:348] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } } peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260619 02:27:37.626878 16238 raft_consensus.cc:348] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } } peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260619 02:27:37.627137 16238 raft_consensus.cc:374] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:37.627211 16238 raft_consensus.cc:729] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b66e078b9f44628e792344f37eb35a, State: Initialized, Role: FOLLOWER
I20260619 02:27:37.627239 16237 raft_consensus.cc:374] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:37.627288 16237 raft_consensus.cc:729] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8fc8a1ef352471b9d3359580371ea24, State: Initialized, Role: FOLLOWER
I20260619 02:27:37.627698 16238 consensus_queue.cc:260] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } } peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260619 02:27:37.627698 16237 consensus_queue.cc:260] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } } peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260619 02:27:37.628360 15951 heartbeater.cc:503] Master 127.0.98.62:40637 was elected leader, sending a full tablet report...
I20260619 02:27:37.628652 16219 heartbeater.cc:503] Master 127.0.98.62:40637 was elected leader, sending a full tablet report...
I20260619 02:27:37.628695 16238 ts_tablet_manager.cc:1428] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:27:37.628733 16237 ts_tablet_manager.cc:1428] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260619 02:27:37.630276 15952 tablet.cc:2369] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:27:37.631201 16236 raft_consensus.cc:348] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } } peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260619 02:27:37.631436 16236 raft_consensus.cc:374] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:37.631510 16236 raft_consensus.cc:729] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ab934f359c4ed7be3cba8cfef0056e, State: Initialized, Role: FOLLOWER
I20260619 02:27:37.631963 16236 consensus_queue.cc:260] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } } peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260619 02:27:37.632611 16085 heartbeater.cc:503] Master 127.0.98.62:40637 was elected leader, sending a full tablet report...
I20260619 02:27:37.632871 16236 ts_tablet_manager.cc:1428] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:27:37.798614 16244 raft_consensus.cc:482] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:27:37.798854 16244 raft_consensus.cc:504] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } } peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260619 02:27:37.800019 16244 leader_election.cc:288] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58b66e078b9f44628e792344f37eb35a (127.0.98.1:36225), c8fc8a1ef352471b9d3359580371ea24 (127.0.98.3:38555)
I20260619 02:27:37.805150 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.800303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33701 (local address 127.0.98.1:36225)
0619 02:27:37.800436 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:37.800443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:37.800723 (+ 280us) server_negotiation.cc:407] Connection header received
0619 02:27:37.800788 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:37.800795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:37.800845 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:37.800918 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:37.801213 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.801981 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.803865 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.804484 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.804598 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.804659 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:37.804944 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 02:27:37.805020 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:27:37.805143 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.800468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57877 (local address 127.0.98.3:38555)
0619 02:27:37.800610 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:37.800614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:37.800625 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:27:37.800670 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:37.800672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:37.800740 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:37.800826 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:37.801142 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.801974 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.803839 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.804484 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.804598 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.804658 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:37.804944 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 02:27:37.805020 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:27:37.805145 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.800240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38555 (local address 127.0.98.2:57877)
0619 02:27:37.800371 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:37.800389 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:37.800481 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:37.800831 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:37.800832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:37.800842 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:37.800999 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.801028 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.802089 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.802092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.803678 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.803688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.804606 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.804609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.804770 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.804806 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:27:37.804884 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:27:37.804993 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":263}
I20260619 02:27:37.805152 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.800244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36225 (local address 127.0.98.2:33701)
0619 02:27:37.800627 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:37.800637 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:37.800723 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:37.800945 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:37.800947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:37.800963 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:37.801061 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.801070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.802089 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.802092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.803678 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:37.803688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.804606 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.804609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:37.804765 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.804815 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 02:27:37.804884 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 02:27:37.804982 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":217,"thread_start_us":122,"threads_started":1}
I20260619 02:27:37.805878 16173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75553a82024e42189bb2591516a33a9e" candidate_uuid: "74ab934f359c4ed7be3cba8cfef0056e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8fc8a1ef352471b9d3359580371ea24" is_pre_election: true
I20260619 02:27:37.805940 15905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75553a82024e42189bb2591516a33a9e" candidate_uuid: "74ab934f359c4ed7be3cba8cfef0056e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58b66e078b9f44628e792344f37eb35a" is_pre_election: true
I20260619 02:27:37.806178 16173 raft_consensus.cc:2454] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74ab934f359c4ed7be3cba8cfef0056e in term 0.
I20260619 02:27:37.806208 15905 raft_consensus.cc:2454] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74ab934f359c4ed7be3cba8cfef0056e in term 0.
I20260619 02:27:37.806702 15973 leader_election.cc:302] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 58b66e078b9f44628e792344f37eb35a, 74ab934f359c4ed7be3cba8cfef0056e, c8fc8a1ef352471b9d3359580371ea24; no voters:
I20260619 02:27:37.806965 16244 raft_consensus.cc:2781] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:27:37.807037 16244 raft_consensus.cc:482] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:27:37.807066 16244 raft_consensus.cc:3037] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:37.808130 16244 raft_consensus.cc:504] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } } peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260619 02:27:37.808461 16244 leader_election.cc:288] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [CANDIDATE]: Term 1 election: Requested vote from peers 58b66e078b9f44628e792344f37eb35a (127.0.98.1:36225), c8fc8a1ef352471b9d3359580371ea24 (127.0.98.3:38555)
I20260619 02:27:37.808780 15905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75553a82024e42189bb2591516a33a9e" candidate_uuid: "74ab934f359c4ed7be3cba8cfef0056e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58b66e078b9f44628e792344f37eb35a"
I20260619 02:27:37.808910 15905 raft_consensus.cc:3037] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a [term 0 FOLLOWER]: Advancing to term 1
W20260619 02:27:37.809701 16220 tablet.cc:2369] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:27:37.810014 15905 raft_consensus.cc:2454] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74ab934f359c4ed7be3cba8cfef0056e in term 1.
I20260619 02:27:37.810276 16173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75553a82024e42189bb2591516a33a9e" candidate_uuid: "74ab934f359c4ed7be3cba8cfef0056e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8fc8a1ef352471b9d3359580371ea24"
I20260619 02:27:37.810335 15972 leader_election.cc:302] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58b66e078b9f44628e792344f37eb35a, 74ab934f359c4ed7be3cba8cfef0056e; no voters:
I20260619 02:27:37.810401 16173 raft_consensus.cc:3037] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:37.810503 16244 raft_consensus.cc:2781] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:27:37.810686 16244 raft_consensus.cc:686] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [term 1 LEADER]: Becoming Leader. State: Replica: 74ab934f359c4ed7be3cba8cfef0056e, State: Running, Role: LEADER
I20260619 02:27:37.811017 16244 consensus_queue.cc:237] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } } peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260619 02:27:37.811594 16173 raft_consensus.cc:2454] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74ab934f359c4ed7be3cba8cfef0056e in term 1.
I20260619 02:27:37.814286 15760 catalog_manager.cc:5495] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e reported cstate change: term changed from 0 to 1, leader changed from <none> to 74ab934f359c4ed7be3cba8cfef0056e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "74ab934f359c4ed7be3cba8cfef0056e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } health_report { overall_health: UNKNOWN } } }
I20260619 02:27:37.835870 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.832681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.0.98.2:44497)
0619 02:27:37.832813 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:37.832816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:37.832829 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:27:37.832898 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:37.832900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:37.832951 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:37.833050 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:27:37.833237 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.833803 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.834624 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:37.834848 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:37.834932 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:37.835156 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:27:37.835558 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:27:37.835645 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:37.835695 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 02:27:37.835741 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:27:37.841361 16173 raft_consensus.cc:1261] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24 [term 1 FOLLOWER]: Refusing update from remote peer 74ab934f359c4ed7be3cba8cfef0056e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:27:37.841361 15905 raft_consensus.cc:1261] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Refusing update from remote peer 74ab934f359c4ed7be3cba8cfef0056e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:27:37.842068 16246 consensus_queue.cc:1035] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:27:37.842329 16244 consensus_queue.cc:1035] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:27:37.848223 16086 tablet.cc:2369] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:27:37.850221 16250 mvcc.cc:205] Tried to move back new op lower bound from 7298400492911423488 to 7298400492797034496. Current Snapshot: MvccSnapshot[applied={T|T < 7298400492911423488}]
I20260619 02:27:37.858160 15758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:27:37.858342 15758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:27:37.860713 15758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Sending DeleteTablet for 3 replicas of tablet 75553a82024e42189bb2591516a33a9e
I20260619 02:27:37.861591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 15822
I20260619 02:27:37.861517 16153 tablet_service.cc:1514] Processing DeleteTablet for tablet 75553a82024e42189bb2591516a33a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59522
I20260619 02:27:37.861588 16019 tablet_service.cc:1514] Processing DeleteTablet for tablet 75553a82024e42189bb2591516a33a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34118
I20260619 02:27:37.862943 16259 tablet_replica.cc:330] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e: stopping tablet replica
I20260619 02:27:37.863252 16259 raft_consensus.cc:2229] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:27:37.863642 16259 raft_consensus.cc:2258] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:27:37.864006 16260 tablet_replica.cc:330] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24: stopping tablet replica
I20260619 02:27:37.864300 16260 raft_consensus.cc:2229] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:27:37.864591 16260 raft_consensus.cc:2258] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:27:37.865962 16260 ts_tablet_manager.cc:1905] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:27:37.866245 16259 ts_tablet_manager.cc:1905] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:27:37.868680 16260 ts_tablet_manager.cc:1918] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:27:37.868806 16259 ts_tablet_manager.cc:1918] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:27:37.868834 16260 log.cc:1196] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/75553a82024e42189bb2591516a33a9e
I20260619 02:27:37.868984 16259 log.cc:1196] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/75553a82024e42189bb2591516a33a9e
I20260619 02:27:37.869153 16260 ts_tablet_manager.cc:1939] T 75553a82024e42189bb2591516a33a9e P c8fc8a1ef352471b9d3359580371ea24: Deleting consensus metadata
I20260619 02:27:37.869534 16259 ts_tablet_manager.cc:1939] T 75553a82024e42189bb2591516a33a9e P 74ab934f359c4ed7be3cba8cfef0056e: Deleting consensus metadata
I20260619 02:27:37.870324 15745 catalog_manager.cc:4841] TS c8fc8a1ef352471b9d3359580371ea24 (127.0.98.3:38555): tablet 75553a82024e42189bb2591516a33a9e (table test-table [id=95b68e0a6ca14f64bf50cd135cf75829]) successfully deleted
I20260619 02:27:37.870777 15746 catalog_manager.cc:4841] TS 74ab934f359c4ed7be3cba8cfef0056e (127.0.98.2:44497): tablet 75553a82024e42189bb2591516a33a9e (table test-table [id=95b68e0a6ca14f64bf50cd135cf75829]) successfully deleted
W20260619 02:27:37.873842 15744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36225 (error 108)
I20260619 02:27:37.874600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 15956
I20260619 02:27:37.874781 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:37.874493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36225 (local address 127.0.0.1:58656)
0619 02:27:37.874603 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:37.874689 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 02:27:37.875289 15744 catalog_manager.cc:4584] TS 58b66e078b9f44628e792344f37eb35a (127.0.98.1:36225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75553a82024e42189bb2591516a33a9e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36225: connect: Connection refused (error 111)
I20260619 02:27:37.880887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 16090
I20260619 02:27:37.886722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 15728
I20260619 02:27:57.892992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40637
--webserver_interface=127.0.98.62
--webserver_port=33015
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:27:58.003576 16262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:58.003854 16262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:58.003896 16262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:58.007874 16262 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:27:58.007931 16262 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:27:58.007947 16262 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:27:58.007961 16262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:27:58.007979 16262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:58.007997 16262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:58.008013 16262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:27:58.008033 16262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:27:58.013345 16262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:58.014441 16262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:58.015411 16262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:58.028344 16269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:58.028409 16268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:58.028371 16271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:58.028803 16262 server_base.cc:1034] running on GCE node
I20260619 02:27:58.029433 16262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:58.030632 16262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:58.031803 16262 hybrid_clock.cc:648] HybridClock initialized: now 1781836078031770 us; error 44 us; skew 500 ppm
Jun 19 02:27:58 dist-test-slave-xkrt krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:58.034747 16262 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:27:58.035704 16262 webserver.cc:460] Webserver started at http://127.0.98.62:33015/ using document root <none> and password file <none>
I20260619 02:27:58.036345 16262 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:58.036427 16262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:58.040462 16262 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:27:58.042778 16278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:58.044123 16262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:27:58.044281 16262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "247d63f901dd4743b8114523694b4402"
format_stamp: "Formatted at 2026-06-19 02:27:36 on dist-test-slave-xkrt"
server_key: "8b76c1e825e2daee14e34086e729d59d"
server_key_iv: "2fafd586f58ca0f64eee3d4b524f8d47"
server_key_version: "encryptionkey@0"
I20260619 02:27:58.044907 16262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:58.059185 16262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:58.062320 16262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:58.062534 16262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:58.069525 16262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40637
I20260619 02:27:58.069867 16330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40637 every 8 connection(s)
I20260619 02:27:58.070777 16262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:27:58.075433 16331 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:27:58.077186 16331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Bootstrap starting.
I20260619 02:27:58.079473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 16262
I20260619 02:27:58.079974 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36225
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40637
--webserver_port=46513
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:27:58.095072 16331 log.cc:824] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:58.103688 16331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:27:58.104110 16331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Bootstrap complete.
I20260619 02:27:58.110324 16331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } }
I20260619 02:27:58.110726 16331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247d63f901dd4743b8114523694b4402, State: Initialized, Role: FOLLOWER
I20260619 02:27:58.111313 16331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } }
I20260619 02:27:58.111435 16331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:27:58.111524 16331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:27:58.111622 16331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:27:58.113517 16331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } }
I20260619 02:27:58.113921 16331 leader_election.cc:302] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 247d63f901dd4743b8114523694b4402; no voters:
I20260619 02:27:58.114284 16331 leader_election.cc:288] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:27:58.114365 16335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:27:58.114565 16335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 2 LEADER]: Becoming Leader. State: Replica: 247d63f901dd4743b8114523694b4402, State: Running, Role: LEADER
I20260619 02:27:58.114851 16335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } }
I20260619 02:27:58.115528 16331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:27:58.116178 16336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "247d63f901dd4743b8114523694b4402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } } }
I20260619 02:27:58.116298 16336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:58.117009 16337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 247d63f901dd4743b8114523694b4402. Latest consensus state: current_term: 2 leader_uuid: "247d63f901dd4743b8114523694b4402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } } }
I20260619 02:27:58.117106 16337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: This master's current role is: LEADER
I20260619 02:27:58.117333 16341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:27:58.121801 16341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:27:58.122437 16341 catalog_manager.cc:1259] Loaded cluster ID: 2b71230bf7b644db8db928f16fc04460
I20260619 02:27:58.122502 16341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:27:58.124995 16341 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:27:58.125703 16341 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Loaded TSK: 0
I20260619 02:27:58.137305 16341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Generated new TSK 1
I20260619 02:27:58.137566 16341 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:27:58.195596 16333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:58.195883 16333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:58.195957 16333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:58.200019 16333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:58.200101 16333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:58.200194 16333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:27:58.205425 16333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46513
--tserver_master_addrs=127.0.98.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:58.206458 16333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:58.207448 16333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:58.217866 16358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:58.217866 16361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:58.218072 16333 server_base.cc:1034] running on GCE node
W20260619 02:27:58.217900 16359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:58.218611 16333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:58.219208 16333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:58.220389 16333 hybrid_clock.cc:648] HybridClock initialized: now 1781836078220383 us; error 33 us; skew 500 ppm
Jun 19 02:27:58 dist-test-slave-xkrt krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:58.223356 16333 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:27:58.224396 16333 webserver.cc:460] Webserver started at http://127.0.98.1:46513/ using document root <none> and password file <none>
I20260619 02:27:58.225109 16333 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:58.225208 16333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:58.230110 16333 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:27:58.232743 16368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:58.234256 16333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:27:58.234380 16333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "58b66e078b9f44628e792344f37eb35a"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "88854bd6dc221055c449acfef2691bb4"
server_key_iv: "ec257d079309e599c1a0fd08648d531a"
server_key_version: "encryptionkey@0"
I20260619 02:27:58.235107 16333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:58.299612 16333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:58.302512 16333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:58.302667 16333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:58.303241 16333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:58.304250 16375 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:27:58.308245 16333 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:27:58.308362 16333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 02:27:58.308441 16333 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:27:58.310140 16333 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:27:58.310211 16333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 02:27:58.310505 16375 tablet_bootstrap.cc:492] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Bootstrap starting.
I20260619 02:27:58.320544 16333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36225
I20260619 02:27:58.320652 16482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36225 every 8 connection(s)
I20260619 02:27:58.321496 16333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:27:58.327488 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 16333
I20260619 02:27:58.327976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44497
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40637
--webserver_port=42995
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:27:58.329777 16375 log.cc:824] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:27:58 dist-test-slave-xkrt krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:58.337529 16375 tablet_bootstrap.cc:492] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:27:58.337958 16375 tablet_bootstrap.cc:492] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Bootstrap complete.
I20260619 02:27:58.339299 16375 ts_tablet_manager.cc:1397] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.004s
I20260619 02:27:58.339715 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.323761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50489 (local address 127.0.98.62:40637)
0619 02:27:58.324784 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:58.324801 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:58.324847 (+ 46us) server_negotiation.cc:407] Connection header received
0619 02:27:58.325588 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:58.325598 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:58.325881 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:58.326207 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:58.327310 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.328455 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.329824 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.330430 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.330623 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.333518 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:58.333527 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:58.333579 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:58.333621 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:58.335947 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:58.336567 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:58.336572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:58.336578 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:58.336718 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:58.337152 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:58.337156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:58.337158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:58.337636 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:58.337774 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:58.338449 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 02:27:58.338753 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":174,"threads_started":1}
I20260619 02:27:58.339789 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.323714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:50489)
0619 02:27:58.324225 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:58.324276 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:58.325165 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:58.326273 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:58.326280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:58.326703 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:58.327143 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:58.327156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.328587 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.328591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:58.329686 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:58.329701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.330589 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.330593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:58.330776 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.331414 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:58.331439 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:58.333291 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:58.336081 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:58.336087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:58.336098 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:58.336434 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:58.336828 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:58.336833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:58.336838 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:58.337033 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:58.337749 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:58.337758 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:58.338090 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 02:27:58.338316 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 02:27:58.338694 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":185,"threads_started":1}
I20260619 02:27:58.342447 16483 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40637
I20260619 02:27:58.342154 16375 raft_consensus.cc:348] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } } peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260619 02:27:58.342572 16375 raft_consensus.cc:729] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b66e078b9f44628e792344f37eb35a, State: Initialized, Role: FOLLOWER
I20260619 02:27:58.342753 16483 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:58.343015 16375 consensus_queue.cc:260] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } } peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260619 02:27:58.343255 16483 heartbeater.cc:511] Master 127.0.98.62:40637 requested a full tablet report, sending...
I20260619 02:27:58.343966 16375 ts_tablet_manager.cc:1428] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:27:58.345873 16295 ts_manager.cc:194] Registered new tserver with Master: 58b66e078b9f44628e792344f37eb35a (127.0.98.1:36225)
I20260619 02:27:58.350165 16295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50489
I20260619 02:27:58.352864 16483 heartbeater.cc:503] Master 127.0.98.62:40637 was elected leader, sending a full tablet report...
Jun 19 02:27:58 dist-test-slave-xkrt krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 02:27:58.358937 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.349144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36225 (local address 127.0.0.1:42096)
0619 02:27:58.349514 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:58.349587 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:58.349706 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:58.350386 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:58.350393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:58.350743 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:58.350882 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:58.350893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.351819 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.351825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:58.352438 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:58.352447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.353052 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.353069 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:58.353203 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.353865 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:58.353878 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:58.355307 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:58.357181 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:58.357192 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:58.357197 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:58.357434 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:58.357758 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:58.357763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:58.357765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:58.357852 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:58.358297 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:58.358303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:58.358432 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 02:27:58.358571 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 02:27:58.358758 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":122,"threads_started":1}
I20260619 02:27:58.358969 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.349187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.0.98.1:36225)
0619 02:27:58.349959 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:58.349967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:58.349984 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:27:58.350035 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:58.350039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:58.350178 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:58.350384 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:58.351028 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.351705 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.352562 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.352902 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.353029 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.355441 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:58.355450 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:58.355463 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:58.355493 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:58.357051 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:58.357569 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:58.357576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:58.357577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:58.357662 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:58.357950 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:58.357953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:58.357955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:58.358181 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:58.358323 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:58.358650 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 02:27:58.358813 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":109,"threads_started":1}
I20260619 02:27:58.360034 16417 tablet_service.cc:1514] Processing DeleteTablet for tablet 75553a82024e42189bb2591516a33a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42096
I20260619 02:27:58.360843 16492 tablet_replica.cc:330] stopping tablet replica
I20260619 02:27:58.361130 16492 raft_consensus.cc:2229] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:27:58.361378 16492 raft_consensus.cc:2258] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:27:58.362365 16492 ts_tablet_manager.cc:1905] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:27:58.366034 16492 ts_tablet_manager.cc:1918] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:27:58.366115 16492 log.cc:1196] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal/wals/75553a82024e42189bb2591516a33a9e
I20260619 02:27:58.366371 16492 ts_tablet_manager.cc:1939] T 75553a82024e42189bb2591516a33a9e P 58b66e078b9f44628e792344f37eb35a: Deleting consensus metadata
I20260619 02:27:58.368193 16279 catalog_manager.cc:4841] TS 58b66e078b9f44628e792344f37eb35a (127.0.98.1:36225): tablet 75553a82024e42189bb2591516a33a9e (table test-table [id=95b68e0a6ca14f64bf50cd135cf75829]) successfully deleted
W20260619 02:27:58.449916 16487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:58.450284 16487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:58.450368 16487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:58.456033 16487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:58.456092 16487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:58.456180 16487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:27:58.463260 16487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42995
--tserver_master_addrs=127.0.98.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:58.464529 16487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:58.465758 16487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:58.475314 16497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:58.475342 16498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:58.475665 16487 server_base.cc:1034] running on GCE node
W20260619 02:27:58.475780 16500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:58.476178 16487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:58.476835 16487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:58.478060 16487 hybrid_clock.cc:648] HybridClock initialized: now 1781836078478025 us; error 42 us; skew 500 ppm
Jun 19 02:27:58 dist-test-slave-xkrt krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:58.481002 16487 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:27:58.482041 16487 webserver.cc:460] Webserver started at http://127.0.98.2:42995/ using document root <none> and password file <none>
I20260619 02:27:58.482637 16487 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:58.482713 16487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:58.487222 16487 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:27:58.489567 16507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:58.490959 16487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:27:58.491104 16487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "74ab934f359c4ed7be3cba8cfef0056e"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "fbad55d1108b6dea1439fb14bba27c67"
server_key_iv: "f95ac784d50c94ca127d0214dfa062f9"
server_key_version: "encryptionkey@0"
I20260619 02:27:58.491693 16487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:58.541009 16487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:58.545855 16487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:58.546087 16487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:58.546902 16487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:58.548005 16487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:58.548053 16487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:58.548120 16487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:58.548153 16487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:58.558454 16487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44497
I20260619 02:27:58.558481 16620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44497 every 8 connection(s)
I20260619 02:27:58.559459 16487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:27:58.566046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 16487
I20260619 02:27:58.566483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38555
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40637
--webserver_port=37489
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:27:58 dist-test-slave-xkrt krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:58.575134 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.561639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34525 (local address 127.0.98.62:40637)
0619 02:27:58.561784 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:58.561788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:58.562627 (+ 839us) server_negotiation.cc:407] Connection header received
0619 02:27:58.564051 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:58.564055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:58.564120 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:58.564235 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:58.565610 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.566283 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.567611 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.567873 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.568004 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.570992 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:58.570995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:58.570998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:58.571024 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:58.572710 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:58.573379 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:58.573382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:58.573384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:58.573462 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:58.573911 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:58.573913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:58.573914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:58.574096 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:58.574173 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:58.574855 (+ 682us) server_negotiation.cc:299] Negotiation successful
0619 02:27:58.574995 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:27:58.576380 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.562026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:34525)
0619 02:27:58.562472 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:58.562518 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:58.563816 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:58.564419 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:58.564426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:58.564930 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:58.565444 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:58.565459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.566416 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.566426 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:58.567475 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:58.567485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.568042 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.568046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:58.568221 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.568854 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:58.568870 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:58.570777 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:58.572835 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:58.572854 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:58.572873 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:58.573262 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:58.573585 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:58.573591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:58.573594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:58.573794 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:58.574204 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:58.574212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:58.574574 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 02:27:58.574809 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 02:27:58.575215 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20260619 02:27:58.577708 16621 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40637
I20260619 02:27:58.577976 16621 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:58.578570 16621 heartbeater.cc:511] Master 127.0.98.62:40637 requested a full tablet report, sending...
I20260619 02:27:58.579658 16295 ts_manager.cc:194] Registered new tserver with Master: 74ab934f359c4ed7be3cba8cfef0056e (127.0.98.2:44497)
I20260619 02:27:58.580322 16295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34525
W20260619 02:27:58.678445 16624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:27:58.678707 16624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:27:58.678783 16624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:27:58.682762 16624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:27:58.682847 16624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:27:58.682938 16624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:27:58.688050 16624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37489
--tserver_master_addrs=127.0.98.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:27:58.689081 16624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:27:58.690140 16624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:27:58.699463 16629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:27:58.699745 16630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:58.699679 16624 server_base.cc:1034] running on GCE node
W20260619 02:27:58.699717 16632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:27:58.700239 16624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:27:58.700836 16624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:27:58.702016 16624 hybrid_clock.cc:648] HybridClock initialized: now 1781836078701997 us; error 34 us; skew 500 ppm
Jun 19 02:27:58 dist-test-slave-xkrt krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:27:58.704715 16624 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:27:58.705528 16624 webserver.cc:460] Webserver started at http://127.0.98.3:37489/ using document root <none> and password file <none>
I20260619 02:27:58.706133 16624 fs_manager.cc:359] Metadata directory not provided
I20260619 02:27:58.706238 16624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:27:58.709859 16624 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 02:27:58.711802 16639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:58.712934 16624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:27:58.713078 16624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "c8fc8a1ef352471b9d3359580371ea24"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "4b7c920e8b674f0bce387abe35eee731"
server_key_iv: "f13518fddf8e465c6d7336f4b7a7c331"
server_key_version: "encryptionkey@0"
I20260619 02:27:58.713626 16624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:27:58.725469 16624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:27:58.728556 16624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:27:58.728751 16624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:27:58.729303 16624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:27:58.730090 16624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:27:58.730161 16624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:58.730225 16624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:27:58.730268 16624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:27:58.739421 16624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38555
I20260619 02:27:58.739465 16752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38555 every 8 connection(s)
I20260619 02:27:58.740450 16624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:27:58.742525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 16624
Jun 19 02:27:58 dist-test-slave-xkrt krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:27:58.751755 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.743118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.0.98.62:40637)
0619 02:27:58.743359 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:58.743364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:58.743661 (+ 297us) server_negotiation.cc:407] Connection header received
0619 02:27:58.743856 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:58.743859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:58.743926 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:58.744049 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:58.744983 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.745670 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.746570 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.746825 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.746937 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.748312 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:58.748318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:58.748321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:58.748349 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:58.750133 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:58.750526 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:58.750530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:58.750531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:58.750622 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:58.750912 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:58.750915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:58.750917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:58.751090 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:58.751314 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:58.751455 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 02:27:58.751571 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":85,"threads_started":1}
I20260619 02:27:58.754578 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.742301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50695 (local address 127.0.98.62:40637)
0619 02:27:58.742455 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:58.742460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:58.743308 (+ 848us) server_negotiation.cc:407] Connection header received
0619 02:27:58.744973 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:58.744976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:58.745024 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:58.745135 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:58.746344 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.746906 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.748220 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.748490 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.748608 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.751112 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:58.751115 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:58.751118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:58.751142 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:58.752778 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:58.753265 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:58.753270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:58.753272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:58.753354 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:58.753726 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:58.753729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:58.753730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:58.753874 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:58.753948 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:58.754380 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 02:27:58.754463 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:27:58.755427 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.742722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:50695)
0619 02:27:58.743164 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:58.743215 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:58.744776 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:58.745311 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:58.745317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:58.745738 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:27:58.746207 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:58.746220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.747029 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.747033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:58.748089 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:58.748103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.748642 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.748646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:58.748826 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.749305 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:27:58.749320 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:27:58.750937 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:27:58.752894 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:58.752899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:58.752912 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:58.753162 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:58.753454 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:27:58.753458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:27:58.753459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:27:58.753650 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:27:58.753964 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:27:58.753968 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:27:58.754174 (+ 206us) client_negotiation.cc:769] Sending connection context
0619 02:27:58.754318 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 02:27:58.754553 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":135,"threads_started":1}
I20260619 02:27:58.756476 16753 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40637
I20260619 02:27:58.756750 16753 heartbeater.cc:464] Registering TS with master...
I20260619 02:27:58.757266 16753 heartbeater.cc:511] Master 127.0.98.62:40637 requested a full tablet report, sending...
I20260619 02:27:58.758350 16295 ts_manager.cc:194] Registered new tserver with Master: c8fc8a1ef352471b9d3359580371ea24 (127.0.98.3:38555)
I20260619 02:27:58.758905 16295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50695
I20260619 02:27:58.765079 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:27:58.769310 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.765664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.0.98.62:40637)
0619 02:27:58.765812 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:58.765817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:58.766008 (+ 191us) server_negotiation.cc:407] Connection header received
0619 02:27:58.766207 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:58.766210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:58.766301 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:58.766411 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:27:58.766672 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.767217 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.768115 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.768349 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.768463 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.768781 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:27:58.768914 (+ 133us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 02:27:58.769175 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35006: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 02:27:58.769366 16486 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35006: token expired
W20260619 02:27:58.769750 16222 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:40637)
I20260619 02:27:58.776993 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.770136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.0.98.62:40637)
0619 02:27:58.770263 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:58.770269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:58.770298 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:27:58.770411 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:58.770414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:58.770459 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:58.770539 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:27:58.771150 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.771615 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.772410 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.772620 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.772719 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.774073 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:27:58.774077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:27:58.774080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:27:58.774110 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:27:58.775369 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:58.775783 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:58.775787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:58.775789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:58.775877 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:27:58.776149 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:27:58.776154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:27:58.776155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:27:58.776321 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:27:58.776465 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:58.776712 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 02:27:58.776823 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:27:58.779450 16295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:27:58.782670 16295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:27:58.790279 16417 tablet_service.cc:1467] Processing CreateTablet for tablet 4e8e64f5152a42cc96bf03fbb81fb354 (DEFAULT_TABLE table=test-table [id=ba41a27998494aacb2dea285210791fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:58.790985 16417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e8e64f5152a42cc96bf03fbb81fb354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:58.794000 16375 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a: Bootstrap starting.
I20260619 02:27:58.794827 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.789291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38555 (local address 127.0.0.1:41132)
0619 02:27:58.789435 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:58.789450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:58.789570 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:58.790561 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:58.790564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:58.790584 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:58.790681 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:58.790687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.792051 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.792054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:58.793271 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:58.793281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.794474 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.794477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:58.794575 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.794594 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:27:58.794636 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 02:27:58.794693 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":23}
I20260619 02:27:58.794827 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.789480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44497 (local address 127.0.0.1:60360)
0619 02:27:58.789994 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:58.790008 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:58.790097 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:58.790892 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:58.790894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:58.790911 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:58.791002 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:58.791008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.791926 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.791931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:58.793327 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:58.793337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.794317 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.794320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:58.794430 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.794483 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 02:27:58.794569 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:27:58.794647 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":137,"threads_started":1}
I20260619 02:27:58.794848 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.789935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.0.98.2:44497)
0619 02:27:58.790317 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:58.790327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:58.790345 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:27:58.790411 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:58.790415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:58.790743 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:58.790921 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:58.791087 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.791807 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.793452 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.794187 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.794298 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.794443 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:58.794604 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 02:27:58.794696 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":101,"threads_started":1}
I20260619 02:27:58.794935 16375 tablet_bootstrap.cc:654] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:58.796689 16375 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a: No bootstrap required, opened a new log
I20260619 02:27:58.796784 16375 ts_tablet_manager.cc:1397] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:27:58.797240 16375 raft_consensus.cc:348] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:27:58.797366 16375 raft_consensus.cc:374] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:58.797405 16375 raft_consensus.cc:729] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b66e078b9f44628e792344f37eb35a, State: Initialized, Role: FOLLOWER
I20260619 02:27:58.797564 16375 consensus_queue.cc:260] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:27:58.797816 16375 ts_tablet_manager.cc:1428] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:27:58.798045 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:58.789502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.0.98.3:38555)
0619 02:27:58.789992 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:58.790004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:58.790038 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:27:58.790118 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:58.790124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:58.790344 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:58.790579 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:58.790766 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.791941 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.793411 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:58.794344 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:58.795772 (+ 1428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:58.797712 (+ 1940us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:58.797766 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 02:27:58.797852 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":156,"threads_started":1}
I20260619 02:27:58.798233 16555 tablet_service.cc:1467] Processing CreateTablet for tablet 4e8e64f5152a42cc96bf03fbb81fb354 (DEFAULT_TABLE table=test-table [id=ba41a27998494aacb2dea285210791fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:58.799403 16555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e8e64f5152a42cc96bf03fbb81fb354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:58.806934 16766 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e: Bootstrap starting.
I20260619 02:27:58.808007 16687 tablet_service.cc:1467] Processing CreateTablet for tablet 4e8e64f5152a42cc96bf03fbb81fb354 (DEFAULT_TABLE table=test-table [id=ba41a27998494aacb2dea285210791fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:27:58.809047 16687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e8e64f5152a42cc96bf03fbb81fb354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:27:58.809615 16766 tablet_bootstrap.cc:654] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:58.810333 16766 log.cc:824] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:58.812196 16766 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e: No bootstrap required, opened a new log
I20260619 02:27:58.812417 16766 ts_tablet_manager.cc:1397] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:27:58.815901 16766 raft_consensus.cc:348] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:27:58.816155 16766 raft_consensus.cc:374] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:58.816215 16766 raft_consensus.cc:729] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ab934f359c4ed7be3cba8cfef0056e, State: Initialized, Role: FOLLOWER
I20260619 02:27:58.816210 16768 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Bootstrap starting.
I20260619 02:27:58.816752 16766 consensus_queue.cc:260] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:27:58.817478 16621 heartbeater.cc:503] Master 127.0.98.62:40637 was elected leader, sending a full tablet report...
I20260619 02:27:58.817874 16766 ts_tablet_manager.cc:1428] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:27:58.818418 16768 tablet_bootstrap.cc:654] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Neither blocks nor log segments found. Creating new log.
I20260619 02:27:58.819453 16768 log.cc:824] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Log is configured to *not* fsync() on all Append() calls
I20260619 02:27:58.821282 16768 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: No bootstrap required, opened a new log
I20260619 02:27:58.821489 16768 ts_tablet_manager.cc:1397] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
W20260619 02:27:58.822686 16484 tablet.cc:2369] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:27:58.824611 16768 raft_consensus.cc:348] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:27:58.824841 16768 raft_consensus.cc:374] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:27:58.824882 16768 raft_consensus.cc:729] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8fc8a1ef352471b9d3359580371ea24, State: Initialized, Role: FOLLOWER
I20260619 02:27:58.825261 16768 consensus_queue.cc:260] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:27:58.825950 16753 heartbeater.cc:503] Master 127.0.98.62:40637 was elected leader, sending a full tablet report...
I20260619 02:27:58.826205 16768 ts_tablet_manager.cc:1428] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
W20260619 02:27:58.991408 16754 tablet.cc:2369] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:27:59.061645 16622 tablet.cc:2369] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:27:59.131867 16769 raft_consensus.cc:482] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:27:59.132087 16769 raft_consensus.cc:504] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:27:59.133297 16769 leader_election.cc:288] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8fc8a1ef352471b9d3359580371ea24 (127.0.98.3:38555), 58b66e078b9f44628e792344f37eb35a (127.0.98.1:36225)
I20260619 02:27:59.137261 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:59.133707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48081 (local address 127.0.98.1:36225)
0619 02:27:59.134005 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:59.134009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:59.134026 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:27:59.134175 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:59.134178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:59.134237 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:59.134354 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:59.134641 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:59.135167 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:59.136273 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:59.136718 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:59.136839 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:59.137015 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:59.137058 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 02:27:59.137125 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
I20260619 02:27:59.137274 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:59.133599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36225 (local address 127.0.98.2:48081)
0619 02:27:59.133919 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:59.133934 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:59.134035 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:59.134364 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:59.134367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:59.134385 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:59.134518 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:59.134524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:59.135283 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:59.135287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:59.136147 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:59.136155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:59.136846 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:59.136849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:59.136947 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:59.136978 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:27:59.137045 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 02:27:59.137116 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":37,"thread_start_us":104,"threads_started":1}
I20260619 02:27:59.137408 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:59.133733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50559 (local address 127.0.98.3:38555)
0619 02:27:59.133878 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:59.133883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:59.134299 (+ 416us) server_negotiation.cc:407] Connection header received
0619 02:27:59.134454 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:59.134459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:59.134508 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:59.134598 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:27:59.134866 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:59.135354 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:59.136417 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:59.136828 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:59.136946 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:59.137007 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:59.137206 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 02:27:59.137274 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:27:59.137573 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:59.133598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38555 (local address 127.0.98.2:50559)
0619 02:27:59.134192 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:59.134204 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:27:59.134339 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:27:59.134606 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:27:59.134609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:27:59.134625 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:27:59.134751 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:59.134757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:59.135470 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:59.135475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:59.136294 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:27:59.136302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:27:59.136953 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:27:59.136956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:27:59.137057 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:59.137086 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 02:27:59.137168 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 02:27:59.137238 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":334,"thread_start_us":52,"threads_started":1}
I20260619 02:27:59.138000 16437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8e64f5152a42cc96bf03fbb81fb354" candidate_uuid: "74ab934f359c4ed7be3cba8cfef0056e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58b66e078b9f44628e792344f37eb35a" is_pre_election: true
I20260619 02:27:59.138208 16707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8e64f5152a42cc96bf03fbb81fb354" candidate_uuid: "74ab934f359c4ed7be3cba8cfef0056e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8fc8a1ef352471b9d3359580371ea24" is_pre_election: true
I20260619 02:27:59.138360 16437 raft_consensus.cc:2454] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74ab934f359c4ed7be3cba8cfef0056e in term 0.
I20260619 02:27:59.138536 16707 raft_consensus.cc:2454] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74ab934f359c4ed7be3cba8cfef0056e in term 0.
I20260619 02:27:59.138839 16508 leader_election.cc:302] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58b66e078b9f44628e792344f37eb35a, 74ab934f359c4ed7be3cba8cfef0056e; no voters:
I20260619 02:27:59.139148 16769 raft_consensus.cc:2781] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:27:59.139225 16769 raft_consensus.cc:482] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:27:59.139256 16769 raft_consensus.cc:3037] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:59.140424 16769 raft_consensus.cc:504] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:27:59.140815 16769 leader_election.cc:288] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [CANDIDATE]: Term 1 election: Requested vote from peers c8fc8a1ef352471b9d3359580371ea24 (127.0.98.3:38555), 58b66e078b9f44628e792344f37eb35a (127.0.98.1:36225)
I20260619 02:27:59.141198 16707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8e64f5152a42cc96bf03fbb81fb354" candidate_uuid: "74ab934f359c4ed7be3cba8cfef0056e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8fc8a1ef352471b9d3359580371ea24"
I20260619 02:27:59.141291 16437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8e64f5152a42cc96bf03fbb81fb354" candidate_uuid: "74ab934f359c4ed7be3cba8cfef0056e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58b66e078b9f44628e792344f37eb35a"
I20260619 02:27:59.141335 16707 raft_consensus.cc:3037] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:59.141462 16437 raft_consensus.cc:3037] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:27:59.142444 16437 raft_consensus.cc:2454] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74ab934f359c4ed7be3cba8cfef0056e in term 1.
I20260619 02:27:59.142449 16707 raft_consensus.cc:2454] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74ab934f359c4ed7be3cba8cfef0056e in term 1.
I20260619 02:27:59.142748 16508 leader_election.cc:302] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58b66e078b9f44628e792344f37eb35a, 74ab934f359c4ed7be3cba8cfef0056e; no voters:
I20260619 02:27:59.142918 16769 raft_consensus.cc:2781] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:27:59.143141 16769 raft_consensus.cc:686] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 1 LEADER]: Becoming Leader. State: Replica: 74ab934f359c4ed7be3cba8cfef0056e, State: Running, Role: LEADER
I20260619 02:27:59.143458 16769 consensus_queue.cc:237] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:27:59.146562 16295 catalog_manager.cc:5495] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e reported cstate change: term changed from 0 to 1, leader changed from <none> to 74ab934f359c4ed7be3cba8cfef0056e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "74ab934f359c4ed7be3cba8cfef0056e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } health_report { overall_health: UNKNOWN } } }
I20260619 02:27:59.160339 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:59.156811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.0.98.2:44497)
0619 02:27:59.156974 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 02:27:59.156979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:27:59.157032 (+ 53us) server_negotiation.cc:407] Connection header received
0619 02:27:59.157218 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:27:59.157220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:27:59.157272 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:27:59.157383 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:27:59.157641 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:59.158134 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:59.158970 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:27:59.159182 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:27:59.159288 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:27:59.159591 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:27:59.159972 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:27:59.160103 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 02:27:59.160150 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 02:27:59.160201 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 02:27:59.166065 16437 raft_consensus.cc:1261] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Refusing update from remote peer 74ab934f359c4ed7be3cba8cfef0056e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:27:59.166069 16707 raft_consensus.cc:1261] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24 [term 1 FOLLOWER]: Refusing update from remote peer 74ab934f359c4ed7be3cba8cfef0056e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:27:59.166744 16769 consensus_queue.cc:1035] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:27:59.166932 16775 consensus_queue.cc:1035] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:27:59.171761 16777 mvcc.cc:205] Tried to move back new op lower bound from 7298400580257378304 to 7298400580174041088. Current Snapshot: MvccSnapshot[applied={T|T < 7298400580257378304}]
I20260619 02:27:59.175690 16779 mvcc.cc:205] Tried to move back new op lower bound from 7298400580257378304 to 7298400580174041088. Current Snapshot: MvccSnapshot[applied={T|T < 7298400580257378304}]
I20260619 02:27:59.180025 16295 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:27:59.180239 16295 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:27:59.182569 16295 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Sending DeleteTablet for 3 replicas of tablet 4e8e64f5152a42cc96bf03fbb81fb354
I20260619 02:27:59.183236 16687 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e8e64f5152a42cc96bf03fbb81fb354 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41132
I20260619 02:27:59.183411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 16333
I20260619 02:27:59.185048 16555 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e8e64f5152a42cc96bf03fbb81fb354 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:27:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60360
W20260619 02:27:59.189142 16279 connection.cc:535] client connection to 127.0.98.1:36225 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 02:27:59.189323 16279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:27:59.189559 16786 tablet_replica.cc:330] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: stopping tablet replica
I20260619 02:27:59.189795 16787 tablet_replica.cc:330] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e: stopping tablet replica
I20260619 02:27:59.189929 16786 raft_consensus.cc:2229] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:27:59.190158 16787 raft_consensus.cc:2229] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:27:59.190482 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:59.190196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36225 (local address 127.0.0.1:42104)
0619 02:27:59.190324 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:59.190398 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 02:27:59.190481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 16487
W20260619 02:27:59.191730 16279 catalog_manager.cc:4584] TS 58b66e078b9f44628e792344f37eb35a (127.0.98.1:36225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e8e64f5152a42cc96bf03fbb81fb354: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36225: connect: Connection refused (error 111)
I20260619 02:27:59.191866 16786 raft_consensus.cc:2258] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:27:59.193295 16786 ts_tablet_manager.cc:1905] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:27:59.195938 16786 ts_tablet_manager.cc:1918] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:27:59.196033 16786 log.cc:1196] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/4e8e64f5152a42cc96bf03fbb81fb354
I20260619 02:27:59.196345 16786 ts_tablet_manager.cc:1939] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Deleting consensus metadata
I20260619 02:27:59.197080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 16624
I20260619 02:27:59.197361 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:27:59.197019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44497 (local address 127.0.0.1:60386)
0619 02:27:59.197205 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 02:27:59.197267 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 02:27:59.204600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 16262
I20260619 02:28:19.212174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40637
--webserver_interface=127.0.98.62
--webserver_port=33015
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:19.321346 16789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:19.321622 16789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:19.321707 16789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:19.325692 16789 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:28:19.325737 16789 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:28:19.325753 16789 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:28:19.325767 16789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:19.325835 16789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:19.325870 16789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:19.325899 16789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:19.325938 16789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:19.331168 16789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:19.332165 16789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:19.333182 16789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:28:19.341696 16789 server_base.cc:1034] running on GCE node
W20260619 02:28:19.341643 16795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:19.341629 16797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:19.341707 16794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:19.342449 16789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:19.343524 16789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:19.344722 16789 hybrid_clock.cc:648] HybridClock initialized: now 1781836099344688 us; error 42 us; skew 500 ppm
Jun 19 02:28:19 dist-test-slave-xkrt krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:19.347440 16789 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:28:19.348327 16789 webserver.cc:460] Webserver started at http://127.0.98.62:33015/ using document root <none> and password file <none>
I20260619 02:28:19.348879 16789 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:19.348969 16789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:19.352722 16789 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:28:19.354844 16804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:19.356060 16789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:28:19.356305 16789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "247d63f901dd4743b8114523694b4402"
format_stamp: "Formatted at 2026-06-19 02:27:36 on dist-test-slave-xkrt"
server_key: "8b76c1e825e2daee14e34086e729d59d"
server_key_iv: "2fafd586f58ca0f64eee3d4b524f8d47"
server_key_version: "encryptionkey@0"
I20260619 02:28:19.356881 16789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:19.374862 16789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:19.378266 16789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:19.378466 16789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:19.385459 16789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40637
I20260619 02:28:19.385761 16856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40637 every 8 connection(s)
I20260619 02:28:19.386636 16789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:19.388226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 16789
I20260619 02:28:19.388728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36225
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40637
--webserver_port=46513
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:28:19.391472 16857 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:28:19.393103 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Bootstrap starting.
I20260619 02:28:19.409770 16857 log.cc:824] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:19.420233 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:28:19.420595 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: Bootstrap complete.
I20260619 02:28:19.424582 16857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } }
I20260619 02:28:19.424962 16857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247d63f901dd4743b8114523694b4402, State: Initialized, Role: FOLLOWER
I20260619 02:28:19.425420 16857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } }
I20260619 02:28:19.425573 16857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:19.425685 16857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:19.425810 16857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 2 FOLLOWER]: Advancing to term 3
I20260619 02:28:19.427531 16857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } }
I20260619 02:28:19.427839 16857 leader_election.cc:302] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 247d63f901dd4743b8114523694b4402; no voters:
I20260619 02:28:19.428131 16857 leader_election.cc:288] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 02:28:19.428263 16861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 02:28:19.428493 16861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [term 3 LEADER]: Becoming Leader. State: Replica: 247d63f901dd4743b8114523694b4402, State: Running, Role: LEADER
I20260619 02:28:19.428788 16861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } }
I20260619 02:28:19.429275 16857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:19.430289 16863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 247d63f901dd4743b8114523694b4402. Latest consensus state: current_term: 3 leader_uuid: "247d63f901dd4743b8114523694b4402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } } }
I20260619 02:28:19.430442 16863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:19.431198 16862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "247d63f901dd4743b8114523694b4402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247d63f901dd4743b8114523694b4402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40637 } } }
I20260619 02:28:19.431298 16862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:19.434118 16878 catalog_manager.cc:1259] Loaded cluster ID: 2b71230bf7b644db8db928f16fc04460
I20260619 02:28:19.434173 16878 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: loading cluster ID for follower catalog manager: success
I20260619 02:28:19.436542 16878 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 247d63f901dd4743b8114523694b4402: acquiring CA information for follower catalog manager: success
F20260619 02:28:19.437727 16878 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdfc7fe19ad google::LogMessage::Fail() at ??:0
@ 0x7fdfc7fe488a google::LogMessage::SendToLog() at ??:0
@ 0x7fdfc7fe14de google::LogMessage::Flush() at ??:0
@ 0x7fdfc7fe2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdfcaca518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fdfd2946989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fdfd2923023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fdfd292354b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fdfd291c652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fdfd291b9e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7fdfd2962541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fdfca224512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdfcb8686db start_thread at ??:0
@ 0x7fdfc6adf61f clone at ??:0
W20260619 02:28:19.515426 16859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:19.515708 16859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:19.515790 16859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:19.519846 16859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:19.519927 16859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:19.520035 16859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:28:19.525286 16859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46513
--tserver_master_addrs=127.0.98.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:19.526345 16859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:19.527335 16859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:19.536847 16899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:19.536852 16901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:19.536844 16898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:19.537739 16859 server_base.cc:1034] running on GCE node
I20260619 02:28:19.538179 16859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:19.538841 16859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:19.540035 16859 hybrid_clock.cc:648] HybridClock initialized: now 1781836099540001 us; error 44 us; skew 500 ppm
Jun 19 02:28:19 dist-test-slave-xkrt krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:19.543203 16859 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:28:19.544301 16859 webserver.cc:460] Webserver started at http://127.0.98.1:46513/ using document root <none> and password file <none>
I20260619 02:28:19.545019 16859 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:19.545112 16859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:19.550084 16859 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 02:28:19.552664 16908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:19.554052 16859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:28:19.554144 16859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "58b66e078b9f44628e792344f37eb35a"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "88854bd6dc221055c449acfef2691bb4"
server_key_iv: "ec257d079309e599c1a0fd08648d531a"
server_key_version: "encryptionkey@0"
I20260619 02:28:19.554734 16859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:19.581513 16859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:19.584800 16859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:19.585006 16859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:19.585667 16859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:19.586867 16915 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:28:19.590816 16859 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:28:19.590875 16859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260619 02:28:19.590921 16859 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:28:19.592644 16859 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:28:19.592700 16859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 02:28:19.592947 16915 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a: Bootstrap starting.
I20260619 02:28:19.603142 16859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36225
I20260619 02:28:19.603354 17022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36225 every 8 connection(s)
I20260619 02:28:19.604313 16859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:28:19.605428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 16859
I20260619 02:28:19.605967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44497
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40637
--webserver_port=42995
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:28:19.614603 16915 log.cc:824] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:19.623534 16915 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:28:19.623956 16915 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a: Bootstrap complete.
I20260619 02:28:19.625373 16915 ts_tablet_manager.cc:1397] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.004s
I20260619 02:28:19.628638 16915 raft_consensus.cc:348] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:28:19.629158 16915 raft_consensus.cc:729] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b66e078b9f44628e792344f37eb35a, State: Initialized, Role: FOLLOWER
I20260619 02:28:19.629704 16915 consensus_queue.cc:260] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:28:19.630714 16915 ts_tablet_manager.cc:1428] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260619 02:28:19.648234 17026 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:28:19.608433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:56725)
0619 02:28:19.609159 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:19.609216 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:19.610264 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:19.647501 (+ 37237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":158,"threads_started":1}
W20260619 02:28:19.648730 17023 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40637 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40637: Client connection negotiation failed: client connection to 127.0.98.62:40637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:28:19.649446 17026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:19.649219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:38615)
0619 02:28:19.649337 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:19.649393 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 02:28:19.650125 17026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:19.649949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:46697)
0619 02:28:19.650039 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:19.650078 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 02:28:19.650354 17023 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:28:19.723436 17025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:19.723665 17025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:19.723706 17025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:19.727656 17025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:19.727705 17025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:19.727815 17025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:28:19.732811 17025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42995
--tserver_master_addrs=127.0.98.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:19.733769 17025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:19.734710 17025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:19.744061 17033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:19.744132 17025 server_base.cc:1034] running on GCE node
W20260619 02:28:19.744078 17036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:19.744079 17034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:19.744699 17025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:19.745286 17025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:19.746440 17025 hybrid_clock.cc:648] HybridClock initialized: now 1781836099746424 us; error 27 us; skew 500 ppm
Jun 19 02:28:19 dist-test-slave-xkrt krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:19.749071 17025 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:28:19.749922 17025 webserver.cc:460] Webserver started at http://127.0.98.2:42995/ using document root <none> and password file <none>
I20260619 02:28:19.750443 17025 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:19.750526 17025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:19.754230 17025 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:28:19.756266 17043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:19.757398 17025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:28:19.757509 17025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "74ab934f359c4ed7be3cba8cfef0056e"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "fbad55d1108b6dea1439fb14bba27c67"
server_key_iv: "f95ac784d50c94ca127d0214dfa062f9"
server_key_version: "encryptionkey@0"
I20260619 02:28:19.758101 17025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:19.772082 17025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:19.774757 17025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:19.774940 17025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:19.775482 17025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:19.776593 17050 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:28:19.780565 17025 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:28:19.780624 17025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 02:28:19.780699 17025 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:28:19.782335 17025 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:28:19.782377 17025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 02:28:19.782663 17050 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e: Bootstrap starting.
I20260619 02:28:19.792743 17025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44497
I20260619 02:28:19.792850 17157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44497 every 8 connection(s)
I20260619 02:28:19.793725 17025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:28:19.794082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 17025
I20260619 02:28:19.794600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38555
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40637
--webserver_port=37489
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:28:19.797250 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:19.795926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:60533)
0619 02:28:19.796417 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:19.796534 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":145,"threads_started":1}
W20260619 02:28:19.797786 17158 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40637 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40637: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
I20260619 02:28:19.798353 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:19.798136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:41911)
0619 02:28:19.798245 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:19.798290 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 02:28:19.798944 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:19.798743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:38967)
0619 02:28:19.798846 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:19.798889 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 02:28:19.799151 17158 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:28:19.801404 17050 log.cc:824] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:19.810531 17050 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:28:19.810921 17050 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e: Bootstrap complete.
I20260619 02:28:19.812394 17050 ts_tablet_manager.cc:1397] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e: Time spent bootstrapping tablet: real 0.030s user 0.019s sys 0.008s
I20260619 02:28:19.816105 17050 raft_consensus.cc:348] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:28:19.816637 17050 raft_consensus.cc:729] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ab934f359c4ed7be3cba8cfef0056e, State: Initialized, Role: FOLLOWER
I20260619 02:28:19.817127 17050 consensus_queue.cc:260] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:28:19.818131 17050 ts_tablet_manager.cc:1428] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260619 02:28:19.818320 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:19.818033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:53965)
0619 02:28:19.818181 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:19.818242 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 02:28:19.818966 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:19.818783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:52573)
0619 02:28:19.818879 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:19.818920 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 02:28:19.910503 17160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:19.910730 17160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:19.910770 17160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:19.914783 17160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:19.914830 17160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:19.914921 17160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:28:19.919873 17160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37489
--tserver_master_addrs=127.0.98.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:19.920814 17160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:19.921841 17160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:19.931346 17171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:19.931349 17168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:19.931350 17169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:19.931456 17160 server_base.cc:1034] running on GCE node
I20260619 02:28:19.932021 17160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:19.932611 17160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:19.933789 17160 hybrid_clock.cc:648] HybridClock initialized: now 1781836099933778 us; error 30 us; skew 500 ppm
Jun 19 02:28:19 dist-test-slave-xkrt krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:19.936256 17160 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:28:19.937034 17160 webserver.cc:460] Webserver started at http://127.0.98.3:37489/ using document root <none> and password file <none>
I20260619 02:28:19.937624 17160 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:19.937714 17160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:19.941177 17160 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:28:19.943116 17178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:19.944257 17160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:28:19.944382 17160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "c8fc8a1ef352471b9d3359580371ea24"
format_stamp: "Formatted at 2026-06-19 02:27:37 on dist-test-slave-xkrt"
server_key: "4b7c920e8b674f0bce387abe35eee731"
server_key_iv: "f13518fddf8e465c6d7336f4b7a7c331"
server_key_version: "encryptionkey@0"
I20260619 02:28:19.944898 17160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:19.976754 17160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:19.979761 17160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:19.979964 17160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:19.980580 17160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:19.981387 17160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:19.981460 17160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:19.981570 17160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:19.981631 17160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:19.990448 17160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38555
I20260619 02:28:19.990477 17291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38555 every 8 connection(s)
I20260619 02:28:19.991395 17160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:28:19.992681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 17160
I20260619 02:28:19.995231 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:19.993582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:42859)
0619 02:28:19.994463 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:19.994558 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":542,"threads_started":1}
W20260619 02:28:19.995656 17292 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40637 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40637: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
I20260619 02:28:19.996169 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:19.995984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:57033)
0619 02:28:19.996079 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:19.996120 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 02:28:19.996749 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:19.996586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:40039)
0619 02:28:19.996670 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:19.996707 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 02:28:19.996956 17292 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:28:20.651402 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:20.650889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:54151)
0619 02:28:20.651200 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:20.651269 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":72,"threads_started":1}
I20260619 02:28:20.820127 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:20.819758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:49631)
0619 02:28:20.819986 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:20.820037 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
I20260619 02:28:20.997870 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:20.997439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:43593)
0619 02:28:20.997722 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:20.997772 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":69,"threads_started":1}
I20260619 02:28:21.082041 17299 raft_consensus.cc:482] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:28:21.082297 17299 raft_consensus.cc:504] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:28:21.083449 17299 leader_election.cc:288] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8fc8a1ef352471b9d3359580371ea24 (127.0.98.3:38555), 74ab934f359c4ed7be3cba8cfef0056e (127.0.98.2:44497)
Jun 19 02:28:21 dist-test-slave-xkrt krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 19 02:28:21 dist-test-slave-xkrt krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 02:28:21.098122 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:21.083657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38555 (local address 127.0.98.1:52087)
0619 02:28:21.083777 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:21.083791 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:21.083893 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:21.085828 (+ 1935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:21.085831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:21.086071 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:21.086558 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:21.086572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:21.089477 (+ 2905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:21.089480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:21.090311 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:21.090321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:21.091260 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:21.091264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:21.091361 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:21.091793 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:28:21.091807 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:21.093457 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:21.095842 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:21.095845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:21.095846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:21.096191 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:21.096563 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:21.096566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:21.096568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:21.096665 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:21.097270 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:21.097272 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:21.097395 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 02:28:21.097594 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 02:28:21.097922 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":29,"spinlock_wait_cycles":12160}
I20260619 02:28:21.098155 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:21.083857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52803 (local address 127.0.98.2:44497)
0619 02:28:21.084189 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:21.084201 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:21.084342 (+ 141us) server_negotiation.cc:407] Connection header received
0619 02:28:21.084609 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:21.084613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:21.084992 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:21.085252 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:21.086729 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:21.087705 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:21.089287 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:21.090064 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:21.090225 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:21.093221 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:21.093228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:21.093245 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:21.093279 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:21.095370 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:21.096028 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:21.096034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:21.096040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:21.096191 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:21.096563 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:21.096566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:21.096568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:21.097031 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:21.097139 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:21.097751 (+ 612us) server_negotiation.cc:299] Negotiation successful
0619 02:28:21.098003 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
I20260619 02:28:21.098219 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:21.083670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44497 (local address 127.0.98.1:52803)
0619 02:28:21.084235 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:21.084248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:21.084367 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:21.085370 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:21.085376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:21.086061 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:21.086560 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:21.086573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:21.087863 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:21.087867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:21.089152 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:21.089163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:21.090187 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:21.090190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:21.090418 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:21.091253 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:28:21.091273 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:21.093042 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:21.095493 (+ 2451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:21.095500 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:21.095514 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:21.095856 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:21.096299 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:21.096302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:21.096304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:21.096464 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:21.097153 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:21.097159 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:21.097405 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 02:28:21.097602 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 02:28:21.098024 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":314,"spinlock_wait_cycles":23936,"thread_start_us":99,"threads_started":1}
I20260619 02:28:21.098811 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:21.083841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52087 (local address 127.0.98.3:38555)
0619 02:28:21.084208 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:21.084231 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:21.084279 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:28:21.085102 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:21.085109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:21.085524 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:21.085859 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:21.087903 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:21.089341 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:21.090435 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:21.091142 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:21.091259 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:21.093717 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:21.093724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:21.093745 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:21.093781 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:21.095700 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:21.096311 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:21.096316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:21.096321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:21.096446 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:21.096754 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:21.096757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:21.096758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:21.097158 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:21.097282 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:21.097711 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 02:28:21.098015 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":142,"threads_started":1}
I20260619 02:28:21.099573 17112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8e64f5152a42cc96bf03fbb81fb354" candidate_uuid: "58b66e078b9f44628e792344f37eb35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74ab934f359c4ed7be3cba8cfef0056e" is_pre_election: true
I20260619 02:28:21.099903 17112 raft_consensus.cc:2454] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58b66e078b9f44628e792344f37eb35a in term 1.
I20260619 02:28:21.100906 16911 leader_election.cc:302] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58b66e078b9f44628e792344f37eb35a, 74ab934f359c4ed7be3cba8cfef0056e; no voters:
W20260619 02:28:21.101141 16910 leader_election.cc:341] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c8fc8a1ef352471b9d3359580371ea24 (127.0.98.3:38555): Not found: Tablet not found: 4e8e64f5152a42cc96bf03fbb81fb354
I20260619 02:28:21.101229 17299 raft_consensus.cc:2781] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 02:28:21.101336 17299 raft_consensus.cc:482] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:28:21.101397 17299 raft_consensus.cc:3037] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:28:21.103539 17299 raft_consensus.cc:504] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:28:21.103982 17299 leader_election.cc:288] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [CANDIDATE]: Term 2 election: Requested vote from peers c8fc8a1ef352471b9d3359580371ea24 (127.0.98.3:38555), 74ab934f359c4ed7be3cba8cfef0056e (127.0.98.2:44497)
I20260619 02:28:21.104349 17112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8e64f5152a42cc96bf03fbb81fb354" candidate_uuid: "58b66e078b9f44628e792344f37eb35a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74ab934f359c4ed7be3cba8cfef0056e"
I20260619 02:28:21.104483 17112 raft_consensus.cc:3037] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 1 FOLLOWER]: Advancing to term 2
W20260619 02:28:21.104538 16910 leader_election.cc:341] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer c8fc8a1ef352471b9d3359580371ea24 (127.0.98.3:38555): Not found: Tablet not found: 4e8e64f5152a42cc96bf03fbb81fb354
I20260619 02:28:21.106420 17112 raft_consensus.cc:2454] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58b66e078b9f44628e792344f37eb35a in term 2.
I20260619 02:28:21.106735 16911 leader_election.cc:302] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58b66e078b9f44628e792344f37eb35a, 74ab934f359c4ed7be3cba8cfef0056e; no voters: c8fc8a1ef352471b9d3359580371ea24
I20260619 02:28:21.106895 17299 raft_consensus.cc:2781] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:28:21.107061 17299 raft_consensus.cc:686] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [term 2 LEADER]: Becoming Leader. State: Replica: 58b66e078b9f44628e792344f37eb35a, State: Running, Role: LEADER
I20260619 02:28:21.107375 17299 consensus_queue.cc:237] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:28:21.107753 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:21.107506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:36665)
0619 02:28:21.107642 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:21.107688 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 02:28:21.571856 17112 raft_consensus.cc:1261] T 4e8e64f5152a42cc96bf03fbb81fb354 P 74ab934f359c4ed7be3cba8cfef0056e [term 2 FOLLOWER]: Refusing update from remote peer 58b66e078b9f44628e792344f37eb35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260619 02:28:21.572683 17299 consensus_queue.cc:1035] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260619 02:28:21.574829 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:21.574423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:33357)
0619 02:28:21.574641 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:21.574706 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
I20260619 02:28:21.577636 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:21.577357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:33051)
0619 02:28:21.577483 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:21.577553 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 02:28:21.577781 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:21.577483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:47039)
0619 02:28:21.577652 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:21.577712 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260619 02:28:21.577776 16910 consensus_peers.cc:487] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a -> Peer c8fc8a1ef352471b9d3359580371ea24 (127.0.98.3:38555): Couldn't send request to peer c8fc8a1ef352471b9d3359580371ea24. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e8e64f5152a42cc96bf03fbb81fb354. This is attempt 1: this message will repeat every 5th retry.
I20260619 02:28:21.974843 17310 ts_tablet_manager.cc:927] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Initiating tablet copy from peer 58b66e078b9f44628e792344f37eb35a (127.0.98.1:36225)
I20260619 02:28:21.975473 17310 tablet_copy_client.cc:316] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:36225
Jun 19 02:28:21 dist-test-slave-xkrt krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 02:28:21.988868 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:21.976017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36225 (local address 127.0.98.3:51081)
0619 02:28:21.976314 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:21.976337 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:21.976489 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:21.976953 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:21.976959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:21.977479 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:21.977661 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:21.977669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:21.978646 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:21.978652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:21.979304 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:21.979312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:21.979910 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:21.979914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:21.980020 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:21.983878 (+ 3858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:28:21.983890 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:21.985334 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:21.987288 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:21.987296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:21.987301 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:21.987538 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:21.987801 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:21.987803 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:21.987804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:21.987887 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:21.988356 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:21.988361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:21.988476 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 02:28:21.988594 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 02:28:21.988729 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":130,"threads_started":1}
I20260619 02:28:21.988933 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:21.976178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51081 (local address 127.0.98.1:36225)
0619 02:28:21.976439 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:21.976447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:21.976472 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:28:21.976620 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:21.976630 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:21.976808 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:21.976956 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:21.977779 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:21.978531 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:21.979430 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:21.979784 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:21.979899 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:21.985464 (+ 5565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:21.985473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:21.985485 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:21.985507 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:21.987177 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:21.987643 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:21.987647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:21.987649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:21.987720 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:21.987996 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:21.987998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:21.987999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:21.988257 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:21.988360 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:21.988671 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 02:28:21.988782 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
I20260619 02:28:21.989991 16997 tablet_copy_service.cc:140] P 58b66e078b9f44628e792344f37eb35a: Received BeginTabletCopySession request for tablet 4e8e64f5152a42cc96bf03fbb81fb354 from peer c8fc8a1ef352471b9d3359580371ea24 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51081)
I20260619 02:28:21.990173 16997 tablet_copy_service.cc:161] P 58b66e078b9f44628e792344f37eb35a: Beginning new tablet copy session on tablet 4e8e64f5152a42cc96bf03fbb81fb354 from peer c8fc8a1ef352471b9d3359580371ea24 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51081: session id = c8fc8a1ef352471b9d3359580371ea24-4e8e64f5152a42cc96bf03fbb81fb354
I20260619 02:28:21.991348 16997 tablet_copy_source_session.cc:205] T 4e8e64f5152a42cc96bf03fbb81fb354 P 58b66e078b9f44628e792344f37eb35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 02:28:21.995498 17310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e8e64f5152a42cc96bf03fbb81fb354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:21.998760 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:21.998504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:55793)
0619 02:28:21.998607 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:21.998662 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 02:28:22.002074 17310 tablet_copy_client.cc:791] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: tablet copy: Starting download of 0 data blocks...
I20260619 02:28:22.002259 17310 tablet_copy_client.cc:655] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: tablet copy: Starting download of 1 WAL segments...
I20260619 02:28:22.003762 17310 tablet_copy_client.cc:531] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:28:22.005378 17310 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Bootstrap starting.
I20260619 02:28:22.010391 17310 log.cc:824] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:22.016520 17310 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:28:22.016760 17310 tablet_bootstrap.cc:492] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Bootstrap complete.
I20260619 02:28:22.017019 17310 ts_tablet_manager.cc:1397] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260619 02:28:22.019170 17310 raft_consensus.cc:348] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:28:22.019389 17310 raft_consensus.cc:729] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8fc8a1ef352471b9d3359580371ea24, State: Initialized, Role: FOLLOWER
I20260619 02:28:22.019729 17310 consensus_queue.cc:260] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fc8a1ef352471b9d3359580371ea24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "74ab934f359c4ed7be3cba8cfef0056e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44497 } } peers { permanent_uuid: "58b66e078b9f44628e792344f37eb35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36225 } }
I20260619 02:28:22.020525 17310 ts_tablet_manager.cc:1428] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20260619 02:28:22.020836 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:22.020562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:58991)
0619 02:28:22.020688 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:22.020743 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 02:28:22.021231 16997 tablet_copy_service.cc:342] P 58b66e078b9f44628e792344f37eb35a: Request end of tablet copy session c8fc8a1ef352471b9d3359580371ea24-4e8e64f5152a42cc96bf03fbb81fb354 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51081
I20260619 02:28:22.021344 16997 tablet_copy_service.cc:433] P 58b66e078b9f44628e792344f37eb35a: ending tablet copy session c8fc8a1ef352471b9d3359580371ea24-4e8e64f5152a42cc96bf03fbb81fb354 on tablet 4e8e64f5152a42cc96bf03fbb81fb354 with peer c8fc8a1ef352471b9d3359580371ea24
I20260619 02:28:22.021524 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:22.021284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:46739)
0619 02:28:22.021375 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:22.021456 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 02:28:22.494207 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:22.493853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:33045)
0619 02:28:22.494064 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:22.494114 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20260619 02:28:22.494374 17246 raft_consensus.cc:1203] T 4e8e64f5152a42cc96bf03fbb81fb354 P c8fc8a1ef352471b9d3359580371ea24 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260619 02:28:22.495270 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:22.495036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:48941)
0619 02:28:22.495153 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:22.495209 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 02:28:22.578775 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:22.578401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:46285)
0619 02:28:22.578629 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:22.578685 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260619 02:28:23.495353 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:23.494946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:58773)
0619 02:28:23.495203 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:23.495259 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
I20260619 02:28:23.496322 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:23.495896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:48805)
0619 02:28:23.496187 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:23.496233 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
I20260619 02:28:23.579912 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:23.579499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:49027)
0619 02:28:23.579766 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:23.579820 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":63,"threads_started":1}
I20260619 02:28:24.496501 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:24.496053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:54567)
0619 02:28:24.496349 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:24.496403 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":69,"threads_started":1}
I20260619 02:28:24.497336 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:24.496977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:48145)
0619 02:28:24.497212 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:24.497262 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
I20260619 02:28:24.581161 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:24.580732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:47781)
0619 02:28:24.580988 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:24.581062 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260619 02:28:25.497942 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:25.497446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:36939)
0619 02:28:25.497797 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:25.497850 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":157,"threads_started":1}
I20260619 02:28:25.498605 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:25.498239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:39901)
0619 02:28:25.498457 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:25.498515 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260619 02:28:25.499073 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:25.498880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:58815)
0619 02:28:25.498979 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:25.499020 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 02:28:25.582484 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:25.582085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:43363)
0619 02:28:25.582333 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:25.582386 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260619 02:28:26.499781 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:26.499284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:47111)
0619 02:28:26.499584 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:26.499655 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":117,"threads_started":1}
I20260619 02:28:26.500077 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:26.499716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:47307)
0619 02:28:26.499927 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:26.499984 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260619 02:28:26.583665 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:26.583278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:34839)
0619 02:28:26.583521 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:26.583571 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260619 02:28:27.500903 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:27.500485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:55013)
0619 02:28:27.500731 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:27.500790 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
I20260619 02:28:27.501232 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:27.500733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:60073)
0619 02:28:27.501063 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:27.501144 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":67,"threads_started":1}
I20260619 02:28:27.584795 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:27.584411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:34249)
0619 02:28:27.584648 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:27.584701 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20260619 02:28:28.502209 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:28.501767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:39589)
0619 02:28:28.502031 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:28.502100 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":131,"threads_started":1}
I20260619 02:28:28.502465 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:28.501971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:57791)
0619 02:28:28.502283 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:28.502352 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":70,"threads_started":1}
I20260619 02:28:28.585985 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:28.585506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:55575)
0619 02:28:28.585832 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:28.585887 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":145,"threads_started":1}
I20260619 02:28:29.503501 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:29.502914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:39809)
0619 02:28:29.503321 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:29.503383 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260619 02:28:29.503556 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:29.503185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:59387)
0619 02:28:29.503409 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:29.503465 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
I20260619 02:28:29.587260 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:29.586846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:46567)
0619 02:28:29.587103 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:29.587158 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260619 02:28:30.504962 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:30.504501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:43423)
0619 02:28:30.504772 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:30.504834 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":110,"threads_started":1}
I20260619 02:28:30.505021 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:30.504569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:55251)
0619 02:28:30.504874 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:30.504927 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":70,"threads_started":1}
I20260619 02:28:30.588603 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:30.588154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:33859)
0619 02:28:30.588438 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:30.588504 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":71,"threads_started":1}
I20260619 02:28:31.506379 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:31.505895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:52561)
0619 02:28:31.506180 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:31.506258 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":94,"threads_started":1}
I20260619 02:28:31.506382 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:31.505854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:40101)
0619 02:28:31.506252 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:31.506301 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":77,"threads_started":1}
I20260619 02:28:31.589923 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:31.589401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:41111)
0619 02:28:31.589747 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:31.589812 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":95,"threads_started":1}
I20260619 02:28:32.507576 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:32.507150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:33973)
0619 02:28:32.507399 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:32.507460 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20260619 02:28:32.507647 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:32.507231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:49599)
0619 02:28:32.507488 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:32.507538 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":65,"threads_started":1}
I20260619 02:28:32.591128 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:32.590689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:58795)
0619 02:28:32.590936 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:32.590999 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260619 02:28:33.508862 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:33.508428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:34389)
0619 02:28:33.508701 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:33.508750 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":107,"threads_started":1}
I20260619 02:28:33.508948 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:33.508386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:37029)
0619 02:28:33.508790 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:33.508840 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":98,"threads_started":1}
I20260619 02:28:33.592325 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:33.591876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:48637)
0619 02:28:33.592130 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:33.592217 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":98,"threads_started":1}
I20260619 02:28:34.510166 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:34.509724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.1:59307)
0619 02:28:34.510014 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:34.510064 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
I20260619 02:28:34.510269 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:34.509751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.3:47787)
0619 02:28:34.510089 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:34.510139 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":69,"threads_started":1}
I20260619 02:28:34.593426 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:34.593047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40637 (local address 127.0.98.2:41995)
0619 02:28:34.593284 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:34.593336 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260619 02:28:35.001159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 16859
I20260619 02:28:35.006948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 17025
I20260619 02:28:35.012543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 17160
2026-06-19T02:28:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59489 ms)
[----------] 30 tests from SecurityITest (241123 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:28:35 dist-test-slave-xkrt krb5kdc[17361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:28:35 dist-test-slave-xkrt krb5kdc[17361](info): set up 2 sockets
Jun 19 02:28:35 dist-test-slave-xkrt krb5kdc[17361](info): commencing operation
krb5kdc: starting...
W20260619 02:28:36.053301 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:28:36 dist-test-slave-xkrt krb5kdc[17361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:28:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:28:36.172662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43201
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:36.285187 17374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:36.285473 17374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:36.285576 17374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:36.289901 17374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:36.289975 17374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:36.290014 17374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:36.290067 17374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:36.290104 17374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:36.295503 17374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:36.296514 17374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:36.297490 17374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:36.305493 17379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:36.305576 17374 server_base.cc:1034] running on GCE node
W20260619 02:28:36.305505 17380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:36.305576 17382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:36.306346 17374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:36.307374 17374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:36.308573 17374 hybrid_clock.cc:648] HybridClock initialized: now 1781836116308547 us; error 41 us; skew 500 ppm
Jun 19 02:28:36 dist-test-slave-xkrt krb5kdc[17361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:36.311335 17374 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:28:36.312148 17374 webserver.cc:460] Webserver started at http://127.0.98.62:37759/ using document root <none> and password file <none>
I20260619 02:28:36.312675 17374 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:36.312762 17374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:36.312986 17374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:36.315037 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "376cd352135a45439c4d3d4584ce5579"
format_stamp: "Formatted at 2026-06-19 02:28:36 on dist-test-slave-xkrt"
server_key: "2a3db6fcbfddec3672338c6b76ed91d6"
server_key_iv: "e526a9ee5b0b5366a0714a2cf759e25d"
server_key_version: "encryptionkey@0"
I20260619 02:28:36.315761 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "376cd352135a45439c4d3d4584ce5579"
format_stamp: "Formatted at 2026-06-19 02:28:36 on dist-test-slave-xkrt"
server_key: "2a3db6fcbfddec3672338c6b76ed91d6"
server_key_iv: "e526a9ee5b0b5366a0714a2cf759e25d"
server_key_version: "encryptionkey@0"
I20260619 02:28:36.319665 17374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:28:36.322079 17389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:36.323338 17374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:28:36.323474 17374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "376cd352135a45439c4d3d4584ce5579"
format_stamp: "Formatted at 2026-06-19 02:28:36 on dist-test-slave-xkrt"
server_key: "2a3db6fcbfddec3672338c6b76ed91d6"
server_key_iv: "e526a9ee5b0b5366a0714a2cf759e25d"
server_key_version: "encryptionkey@0"
I20260619 02:28:36.323589 17374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:36.348568 17374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:36.351610 17374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:36.351822 17374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:36.358978 17374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43201
I20260619 02:28:36.359001 17441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43201 every 8 connection(s)
I20260619 02:28:36.359906 17374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:36.362712 17442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:36.368100 17442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579: Bootstrap starting.
I20260619 02:28:36.368609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 17374
I20260619 02:28:36.368741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:36.369043 392 external_mini_cluster.cc:1428] Setting key 00179cd695f7c61c5819a6415cc7bbfc
I20260619 02:28:36.370935 17442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:36.371655 17442 log.cc:824] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:36.373885 17442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579: No bootstrap required, opened a new log
Jun 19 02:28:36 dist-test-slave-xkrt krb5kdc[17361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:28:36.378446 17442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376cd352135a45439c4d3d4584ce5579" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260619 02:28:36.378700 17442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:36.378777 17442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376cd352135a45439c4d3d4584ce5579, State: Initialized, Role: FOLLOWER
I20260619 02:28:36.379232 17442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376cd352135a45439c4d3d4584ce5579" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260619 02:28:36.379384 17442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:36.379473 17442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:36.379601 17442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:36.380769 17442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376cd352135a45439c4d3d4584ce5579" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260619 02:28:36.381138 17442 leader_election.cc:302] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 376cd352135a45439c4d3d4584ce5579; no voters:
I20260619 02:28:36.381500 17442 leader_election.cc:288] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:36.381580 17447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:36.381793 17447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [term 1 LEADER]: Becoming Leader. State: Replica: 376cd352135a45439c4d3d4584ce5579, State: Running, Role: LEADER
I20260619 02:28:36.382141 17447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376cd352135a45439c4d3d4584ce5579" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260619 02:28:36.382783 17442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:36.383870 17448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "376cd352135a45439c4d3d4584ce5579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376cd352135a45439c4d3d4584ce5579" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } } }
I20260619 02:28:36.384007 17448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:36.384274 17449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 376cd352135a45439c4d3d4584ce5579. Latest consensus state: current_term: 1 leader_uuid: "376cd352135a45439c4d3d4584ce5579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376cd352135a45439c4d3d4584ce5579" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } } }
I20260619 02:28:36.384354 17456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:36.384459 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:36.370421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.0.98.62:43201)
0619 02:28:36.370832 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:36.370846 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:36.370879 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:28:36.371537 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:36.371554 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:36.371918 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:36.372288 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:36.373116 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:36.374322 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:36.375523 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:36.375962 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:36.376074 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:36.378884 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:36.378895 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:36.378907 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:36.378932 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:36.381204 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:36.381761 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:36.381766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:36.381770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:36.381935 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:36.382229 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:36.382232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:36.382236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:36.382596 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:36.382818 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:36.383081 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 02:28:36.383390 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260619 02:28:36.384433 17449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:36.387565 17456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:36.393762 17456 catalog_manager.cc:1347] Generated new cluster ID: bec1f2eea884455c8c719fced809f237
I20260619 02:28:36.393834 17456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:36.404819 17456 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:36.405762 17456 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:36.414526 17456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579: Generated new TSK 0
I20260619 02:28:36.415278 17456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:28:36.452522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43201
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:36.567571 17468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:36.567804 17468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:36.567885 17468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:36.571908 17468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:36.571959 17468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:36.572044 17468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:28:36.577076 17468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:36.578060 17468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:36.578997 17468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:36.588289 17474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:36.588306 17473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:36.588333 17468 server_base.cc:1034] running on GCE node
W20260619 02:28:36.588287 17476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:36.588860 17468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:36.589452 17468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:36.590616 17468 hybrid_clock.cc:648] HybridClock initialized: now 1781836116590598 us; error 27 us; skew 500 ppm
Jun 19 02:28:36 dist-test-slave-xkrt krb5kdc[17361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:36.593176 17468 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:28:36.594038 17468 webserver.cc:460] Webserver started at http://127.0.98.1:45455/ using document root <none> and password file <none>
I20260619 02:28:36.594574 17468 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:36.594645 17468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:36.594889 17468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:36.596818 17468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "f5836da58aa94012b837968cb1c58a23"
format_stamp: "Formatted at 2026-06-19 02:28:36 on dist-test-slave-xkrt"
server_key: "5a0272aca35a0b755220f6e601e7556e"
server_key_iv: "8ac98153b650c2298d85ba857875e234"
server_key_version: "encryptionkey@0"
I20260619 02:28:36.597357 17468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f5836da58aa94012b837968cb1c58a23"
format_stamp: "Formatted at 2026-06-19 02:28:36 on dist-test-slave-xkrt"
server_key: "5a0272aca35a0b755220f6e601e7556e"
server_key_iv: "8ac98153b650c2298d85ba857875e234"
server_key_version: "encryptionkey@0"
I20260619 02:28:36.600970 17468 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:28:36.603245 17483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:36.604441 17468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 02:28:36.604532 17468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "f5836da58aa94012b837968cb1c58a23"
format_stamp: "Formatted at 2026-06-19 02:28:36 on dist-test-slave-xkrt"
server_key: "5a0272aca35a0b755220f6e601e7556e"
server_key_iv: "8ac98153b650c2298d85ba857875e234"
server_key_version: "encryptionkey@0"
I20260619 02:28:36.604625 17468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:36.640972 17468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:36.643712 17468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:36.643862 17468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:36.644392 17468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:36.645151 17468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:36.645197 17468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:36.645237 17468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:36.645257 17468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:36.653940 17468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34757
I20260619 02:28:36.653967 17596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34757 every 8 connection(s)
I20260619 02:28:36.654850 17468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:28:36.659310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 17468
I20260619 02:28:36.659441 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:28:36.659731 392 external_mini_cluster.cc:1428] Setting key 702858868970215f780adccc2bcd7f44
Jun 19 02:28:36 dist-test-slave-xkrt krb5kdc[17361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:28:36.669436 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:36.656598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47845 (local address 127.0.98.62:43201)
0619 02:28:36.656742 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:36.656746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:36.657559 (+ 813us) server_negotiation.cc:407] Connection header received
0619 02:28:36.658834 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:36.658837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:36.658891 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:36.658997 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:36.660413 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:36.660997 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:36.662755 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:36.663168 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:36.663285 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:36.665645 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:36.665652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:36.665659 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:36.665692 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:36.667357 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:36.667847 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:36.667853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:36.667857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:36.667951 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:36.668283 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:36.668288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:36.668293 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:36.668541 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:36.668657 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:36.669130 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 02:28:36.669266 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:28:36.670214 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:36.656964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:47845)
0619 02:28:36.657392 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:36.657435 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:36.658630 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:36.659211 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:36.659219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:36.659657 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:36.660263 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:36.660293 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:36.661129 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:36.661133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:36.662629 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:36.662644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:36.663324 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:36.663327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:36.663462 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:36.664060 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:28:36.664076 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:36.665457 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:36.667480 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:36.667484 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:36.667494 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:36.667734 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:36.668045 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:36.668047 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:36.668049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:36.668167 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:36.668653 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:36.668657 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:36.668873 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 02:28:36.669015 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 02:28:36.669260 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
I20260619 02:28:36.671483 17597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43201
I20260619 02:28:36.671751 17597 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 02:28:36.672305 17597 heartbeater.cc:511] Master 127.0.98.62:43201 requested a full tablet report, sending...
I20260619 02:28:36.673879 17406 ts_manager.cc:194] Registered new tserver with Master: f5836da58aa94012b837968cb1c58a23 (127.0.98.1:34757)
I20260619 02:28:36.674966 17406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:28:36.688709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43201
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:36.795784 17602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:36.796020 17602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:36.796061 17602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:36.800012 17602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:36.800067 17602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:36.800125 17602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:28:36.805169 17602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:36.806147 17602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:36.807078 17602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:36.816361 17607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:36.816366 17608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:36.816433 17602 server_base.cc:1034] running on GCE node
W20260619 02:28:36.816366 17610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:36.816969 17602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:36.817644 17602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:36.818867 17602 hybrid_clock.cc:648] HybridClock initialized: now 1781836116818851 us; error 28 us; skew 500 ppm
Jun 19 02:28:36 dist-test-slave-xkrt krb5kdc[17361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:36.821867 17602 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:28:36.822796 17602 webserver.cc:460] Webserver started at http://127.0.98.2:43507/ using document root <none> and password file <none>
I20260619 02:28:36.823406 17602 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:36.823508 17602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:36.823750 17602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:36.826040 17602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "d349ba42f73848e2afddb406b27b4ee0"
format_stamp: "Formatted at 2026-06-19 02:28:36 on dist-test-slave-xkrt"
server_key: "5479da4ee301947a454f4538a92e548f"
server_key_iv: "486567db6da2e518afe61cc85ac9f3bb"
server_key_version: "encryptionkey@0"
I20260619 02:28:36.826668 17602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d349ba42f73848e2afddb406b27b4ee0"
format_stamp: "Formatted at 2026-06-19 02:28:36 on dist-test-slave-xkrt"
server_key: "5479da4ee301947a454f4538a92e548f"
server_key_iv: "486567db6da2e518afe61cc85ac9f3bb"
server_key_version: "encryptionkey@0"
I20260619 02:28:36.830695 17602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:28:36.833382 17617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:36.834653 17602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:28:36.834784 17602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "d349ba42f73848e2afddb406b27b4ee0"
format_stamp: "Formatted at 2026-06-19 02:28:36 on dist-test-slave-xkrt"
server_key: "5479da4ee301947a454f4538a92e548f"
server_key_iv: "486567db6da2e518afe61cc85ac9f3bb"
server_key_version: "encryptionkey@0"
I20260619 02:28:36.834896 17602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:36.850481 17602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:36.853302 17602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:36.853507 17602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:36.854223 17602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:36.854998 17602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:36.855070 17602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:36.855146 17602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:36.855198 17602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:36.863488 17602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33649
I20260619 02:28:36.863511 17730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33649 every 8 connection(s)
I20260619 02:28:36.864327 17602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:28:36 dist-test-slave-xkrt krb5kdc[17361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:28:36.874392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 17602
I20260619 02:28:36.874536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:28:36.874886 392 external_mini_cluster.cc:1428] Setting key 7e53f064c92bbe506f656f1283047ea5
I20260619 02:28:36.877373 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:36.866071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59105 (local address 127.0.98.62:43201)
0619 02:28:36.866221 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:36.866224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:36.866851 (+ 627us) server_negotiation.cc:407] Connection header received
0619 02:28:36.868180 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:36.868184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:36.868243 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:36.868320 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:36.869454 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:36.870051 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:36.871188 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:36.871417 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:36.871529 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:36.873896 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:36.873900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:36.873902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:36.873924 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:36.875381 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:36.875888 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:36.875891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:36.875893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:36.875968 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:36.876307 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:36.876312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:36.876313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:36.876487 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:36.876644 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:36.877107 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 02:28:36.877222 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:28:36.878284 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:36.866289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:59105)
0619 02:28:36.866690 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:36.866730 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:36.867951 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:36.868483 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:36.868490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:36.868824 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:36.869287 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:36.869300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:36.870180 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:36.870184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:36.871044 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:36.871053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:36.871568 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:36.871572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:36.871741 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:36.872330 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:28:36.872346 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:36.873727 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:36.875497 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:36.875502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:36.875516 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:36.875767 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:36.876066 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:36.876070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:36.876072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:36.876211 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:36.876639 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:36.876645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:36.876880 (+ 235us) client_negotiation.cc:769] Sending connection context
0619 02:28:36.877063 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 02:28:36.877375 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":108,"threads_started":1}
I20260619 02:28:36.879402 17731 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43201
I20260619 02:28:36.879640 17731 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:36.880215 17731 heartbeater.cc:511] Master 127.0.98.62:43201 requested a full tablet report, sending...
I20260619 02:28:36.881332 17406 ts_manager.cc:194] Registered new tserver with Master: d349ba42f73848e2afddb406b27b4ee0 (127.0.98.2:33649)
I20260619 02:28:36.882193 17406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59105
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:28:36.902047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43201
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:37.008142 17736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:37.008405 17736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:37.008473 17736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:37.012534 17736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:37.012615 17736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:37.012704 17736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:28:37.017856 17736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:37.018803 17736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:37.019733 17736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:37.029204 17742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:37.029433 17744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:37.029318 17736 server_base.cc:1034] running on GCE node
W20260619 02:28:37.029285 17741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:37.029968 17736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:37.030591 17736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:37.031780 17736 hybrid_clock.cc:648] HybridClock initialized: now 1781836117031750 us; error 41 us; skew 500 ppm
Jun 19 02:28:37 dist-test-slave-xkrt krb5kdc[17361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:37.034487 17736 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:28:37.035356 17736 webserver.cc:460] Webserver started at http://127.0.98.3:35555/ using document root <none> and password file <none>
I20260619 02:28:37.035892 17736 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:37.035949 17736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:37.036100 17736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:37.038069 17736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "a808fc21ca284713976e332909983ad9"
format_stamp: "Formatted at 2026-06-19 02:28:37 on dist-test-slave-xkrt"
server_key: "1f2cb28afbea372f35aee0c962b9d4f4"
server_key_iv: "ce4bfd32378505527240f09641d15acb"
server_key_version: "encryptionkey@0"
I20260619 02:28:37.038556 17736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a808fc21ca284713976e332909983ad9"
format_stamp: "Formatted at 2026-06-19 02:28:37 on dist-test-slave-xkrt"
server_key: "1f2cb28afbea372f35aee0c962b9d4f4"
server_key_iv: "ce4bfd32378505527240f09641d15acb"
server_key_version: "encryptionkey@0"
I20260619 02:28:37.042052 17736 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:28:37.044337 17751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:37.045456 17736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:28:37.045629 17736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "a808fc21ca284713976e332909983ad9"
format_stamp: "Formatted at 2026-06-19 02:28:37 on dist-test-slave-xkrt"
server_key: "1f2cb28afbea372f35aee0c962b9d4f4"
server_key_iv: "ce4bfd32378505527240f09641d15acb"
server_key_version: "encryptionkey@0"
I20260619 02:28:37.045745 17736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:37.074399 17736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:37.077793 17736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:37.077997 17736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:37.078579 17736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:37.079383 17736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:37.079453 17736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:37.079525 17736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:37.079577 17736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:37.089210 17736 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36681
I20260619 02:28:37.089253 17864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36681 every 8 connection(s)
I20260619 02:28:37.090176 17736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:28:37.097688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 17736
I20260619 02:28:37.097822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:28:37.098114 392 external_mini_cluster.cc:1428] Setting key 350698a0d1c01d051f84cae34893fede
Jun 19 02:28:37 dist-test-slave-xkrt krb5kdc[17361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:28:37.104528 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.092053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56929 (local address 127.0.98.62:43201)
0619 02:28:37.092199 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:37.092208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:37.093084 (+ 876us) server_negotiation.cc:407] Connection header received
0619 02:28:37.094666 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:37.094673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:37.094745 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:37.094852 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:37.096072 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.096612 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.097967 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.098234 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.098319 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.101169 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:37.101172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:37.101174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:37.101198 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:37.102729 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:37.103213 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:37.103216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:37.103217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:37.103291 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:37.103574 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:37.103576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:37.103577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:37.103756 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:37.103830 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:37.104291 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 02:28:37.104412 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:28:37.105290 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.092485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:56929)
0619 02:28:37.092921 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:37.092963 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:37.094372 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:37.095054 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:37.095062 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:37.095466 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:37.095931 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:37.095943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.096724 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.096727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:37.097809 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:37.097820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.098383 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.098387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:37.098576 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.099312 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:28:37.099332 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:37.101004 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:37.102852 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:37.102858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:37.102872 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:37.103128 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:37.103371 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:37.103373 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:37.103374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:37.103500 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:37.103840 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:37.103846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:37.104073 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 02:28:37.104227 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 02:28:37.104467 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
I20260619 02:28:37.106431 17865 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43201
I20260619 02:28:37.106659 17865 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:37.107128 17865 heartbeater.cc:511] Master 127.0.98.62:43201 requested a full tablet report, sending...
I20260619 02:28:37.108134 17406 ts_manager.cc:194] Registered new tserver with Master: a808fc21ca284713976e332909983ad9 (127.0.98.3:36681)
I20260619 02:28:37.108690 17406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56929
I20260619 02:28:37.111923 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:28:37.121025 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.113772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.0.98.62:43201)
0619 02:28:37.113905 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:37.113911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:37.114117 (+ 206us) server_negotiation.cc:407] Connection header received
0619 02:28:37.114280 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:37.114283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:37.114325 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:37.114408 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:37.115091 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.115587 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.116411 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.116678 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.116779 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.118176 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:37.118180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:37.118183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:37.118206 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:37.119414 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:37.119819 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:37.119823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:37.119825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:37.119906 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:37.120170 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:37.120175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:37.120176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:37.120383 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:37.120519 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:37.120756 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 02:28:37.120868 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:28:37.123684 17406 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:28:37.126609 17406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:28:37.140223 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.134433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34757 (local address 127.0.0.1:55408)
0619 02:28:37.134827 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:37.134879 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:37.134998 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:37.135757 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:37.135764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:37.135789 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:37.135925 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:37.135935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.137068 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.137074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:37.138281 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:37.138290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.139809 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.139813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:37.139928 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.139970 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 02:28:37.140038 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 02:28:37.140116 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":62,"threads_started":1}
I20260619 02:28:37.140867 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.135135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33649 (local address 127.0.0.1:37872)
0619 02:28:37.135413 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:37.135426 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:37.135514 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:37.136529 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:37.136532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:37.136550 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:37.136626 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:37.136632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.138060 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.138063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:37.139293 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:37.139303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.140410 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.140414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:37.140552 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.140577 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:28:37.140638 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 02:28:37.140710 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":49,"thread_start_us":86,"threads_started":1}
I20260619 02:28:37.141036 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.134564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55408 (local address 127.0.98.1:34757)
0619 02:28:37.134975 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:37.134990 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:37.135013 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:28:37.135205 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:37.135212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:37.135420 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:37.135570 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:37.136066 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.136954 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.138409 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.139619 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.140569 (+ 950us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.140721 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:37.140771 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 02:28:37.140861 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":120,"threads_started":1}
I20260619 02:28:37.141031 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.135789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.0.98.2:33649)
0619 02:28:37.136091 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:37.136106 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:37.136130 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:28:37.136204 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:37.136209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:37.136387 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:37.136537 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:37.136727 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.137898 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.139432 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.140266 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.140554 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.140735 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:37.140785 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 02:28:37.140867 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":117,"threads_started":1}
I20260619 02:28:37.140902 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.134495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36681 (local address 127.0.0.1:60826)
0619 02:28:37.134864 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:37.134881 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:37.135015 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:37.136638 (+ 1623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:37.136641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:37.136657 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:37.136757 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:37.136772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.138307 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.138310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:37.139430 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:37.139437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.140489 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.140492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:37.140593 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.140614 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:28:37.140682 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 02:28:37.140751 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":5,"thread_start_us":98,"threads_started":1}
I20260619 02:28:37.141630 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.135363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.0.98.3:36681)
0619 02:28:37.136249 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:37.136263 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:37.136281 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:28:37.136340 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:37.136344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:37.136497 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:37.136903 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:37.137042 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.138173 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.139611 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.140365 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.141002 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.141326 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:37.141376 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 02:28:37.141473 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":147,"threads_started":1}
I20260619 02:28:37.143915 17799 tablet_service.cc:1467] Processing CreateTablet for tablet 290f132f338441f2bd8741dc58636682 (DEFAULT_TABLE table=test-table [id=5a5a277ba12b4a589239e36cf452271a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:37.143966 17531 tablet_service.cc:1467] Processing CreateTablet for tablet 290f132f338441f2bd8741dc58636682 (DEFAULT_TABLE table=test-table [id=5a5a277ba12b4a589239e36cf452271a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:37.143968 17665 tablet_service.cc:1467] Processing CreateTablet for tablet 290f132f338441f2bd8741dc58636682 (DEFAULT_TABLE table=test-table [id=5a5a277ba12b4a589239e36cf452271a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:37.144896 17799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290f132f338441f2bd8741dc58636682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:37.144950 17531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290f132f338441f2bd8741dc58636682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:37.144950 17665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290f132f338441f2bd8741dc58636682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:37.151201 17882 tablet_bootstrap.cc:492] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9: Bootstrap starting.
I20260619 02:28:37.152457 17884 tablet_bootstrap.cc:492] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0: Bootstrap starting.
I20260619 02:28:37.152503 17883 tablet_bootstrap.cc:492] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23: Bootstrap starting.
I20260619 02:28:37.153364 17882 tablet_bootstrap.cc:654] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:37.154495 17882 log.cc:824] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:37.154575 17883 tablet_bootstrap.cc:654] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:37.154999 17884 tablet_bootstrap.cc:654] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:37.155838 17883 log.cc:824] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:37.155838 17884 log.cc:824] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:37.156575 17882 tablet_bootstrap.cc:492] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9: No bootstrap required, opened a new log
I20260619 02:28:37.156744 17882 ts_tablet_manager.cc:1397] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:28:37.157492 17883 tablet_bootstrap.cc:492] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23: No bootstrap required, opened a new log
I20260619 02:28:37.157711 17883 ts_tablet_manager.cc:1397] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:28:37.158744 17884 tablet_bootstrap.cc:492] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0: No bootstrap required, opened a new log
I20260619 02:28:37.158928 17884 ts_tablet_manager.cc:1397] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 02:28:37.160356 17882 raft_consensus.cc:348] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5836da58aa94012b837968cb1c58a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34757 } } peers { permanent_uuid: "a808fc21ca284713976e332909983ad9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } } peers { permanent_uuid: "d349ba42f73848e2afddb406b27b4ee0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33649 } }
I20260619 02:28:37.160461 17883 raft_consensus.cc:348] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5836da58aa94012b837968cb1c58a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34757 } } peers { permanent_uuid: "a808fc21ca284713976e332909983ad9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } } peers { permanent_uuid: "d349ba42f73848e2afddb406b27b4ee0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33649 } }
I20260619 02:28:37.160640 17882 raft_consensus.cc:374] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:37.160668 17883 raft_consensus.cc:374] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:37.160733 17882 raft_consensus.cc:729] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a808fc21ca284713976e332909983ad9, State: Initialized, Role: FOLLOWER
I20260619 02:28:37.160748 17883 raft_consensus.cc:729] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5836da58aa94012b837968cb1c58a23, State: Initialized, Role: FOLLOWER
I20260619 02:28:37.161166 17883 consensus_queue.cc:260] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5836da58aa94012b837968cb1c58a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34757 } } peers { permanent_uuid: "a808fc21ca284713976e332909983ad9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } } peers { permanent_uuid: "d349ba42f73848e2afddb406b27b4ee0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33649 } }
I20260619 02:28:37.161202 17882 consensus_queue.cc:260] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5836da58aa94012b837968cb1c58a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34757 } } peers { permanent_uuid: "a808fc21ca284713976e332909983ad9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } } peers { permanent_uuid: "d349ba42f73848e2afddb406b27b4ee0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33649 } }
I20260619 02:28:37.161804 17597 heartbeater.cc:503] Master 127.0.98.62:43201 was elected leader, sending a full tablet report...
I20260619 02:28:37.162155 17883 ts_tablet_manager.cc:1428] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 02:28:37.162339 17882 ts_tablet_manager.cc:1428] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 02:28:37.162688 17865 heartbeater.cc:503] Master 127.0.98.62:43201 was elected leader, sending a full tablet report...
I20260619 02:28:37.163076 17884 raft_consensus.cc:348] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5836da58aa94012b837968cb1c58a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34757 } } peers { permanent_uuid: "a808fc21ca284713976e332909983ad9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } } peers { permanent_uuid: "d349ba42f73848e2afddb406b27b4ee0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33649 } }
I20260619 02:28:37.163309 17884 raft_consensus.cc:374] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:37.163357 17884 raft_consensus.cc:729] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d349ba42f73848e2afddb406b27b4ee0, State: Initialized, Role: FOLLOWER
I20260619 02:28:37.163726 17884 consensus_queue.cc:260] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5836da58aa94012b837968cb1c58a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34757 } } peers { permanent_uuid: "a808fc21ca284713976e332909983ad9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } } peers { permanent_uuid: "d349ba42f73848e2afddb406b27b4ee0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33649 } }
I20260619 02:28:37.164346 17731 heartbeater.cc:503] Master 127.0.98.62:43201 was elected leader, sending a full tablet report...
I20260619 02:28:37.164582 17884 ts_tablet_manager.cc:1428] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 02:28:37.341156 17866 tablet.cc:2369] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:28:37.365422 17732 tablet.cc:2369] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:28:37.406268 17598 tablet.cc:2369] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:28:37.533293 17889 raft_consensus.cc:482] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:28:37.533565 17889 raft_consensus.cc:504] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5836da58aa94012b837968cb1c58a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34757 } } peers { permanent_uuid: "a808fc21ca284713976e332909983ad9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } } peers { permanent_uuid: "d349ba42f73848e2afddb406b27b4ee0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33649 } }
I20260619 02:28:37.534723 17889 leader_election.cc:288] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5836da58aa94012b837968cb1c58a23 (127.0.98.1:34757), d349ba42f73848e2afddb406b27b4ee0 (127.0.98.2:33649)
I20260619 02:28:37.539059 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.534972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33649 (local address 127.0.98.3:38931)
0619 02:28:37.535129 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:37.535144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:37.535269 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:37.535608 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:37.535611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:37.535629 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:37.535756 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:37.535762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.536497 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.536500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:37.537797 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:37.537807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.538616 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.538618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:37.538742 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.538769 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 02:28:37.538846 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 02:28:37.538922 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 02:28:37.539099 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.535066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38931 (local address 127.0.98.2:33649)
0619 02:28:37.535226 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:37.535233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:37.535254 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:28:37.535409 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:37.535411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:37.535485 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:37.535595 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:37.535867 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.536387 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.537908 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.538495 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.538607 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.538697 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:37.538883 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 02:28:37.538973 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:28:37.539116 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.534974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34757 (local address 127.0.98.3:45065)
0619 02:28:37.535415 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:37.535426 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:37.535533 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:37.535818 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:37.535821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:37.535837 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:37.535956 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:37.535962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.536789 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.536792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:37.537953 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:37.537960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.538768 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.538771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:37.538874 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.538892 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 02:28:37.538955 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 02:28:37.539006 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":131,"thread_start_us":121,"threads_started":1}
I20260619 02:28:37.539186 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.535042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45065 (local address 127.0.98.1:34757)
0619 02:28:37.535165 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:37.535172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:37.535509 (+ 337us) server_negotiation.cc:407] Connection header received
0619 02:28:37.535666 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:37.535669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:37.535725 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:37.535809 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:37.536074 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.536674 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.538052 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.538634 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.538755 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.538808 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:37.538981 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 02:28:37.539046 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:28:37.539844 17551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290f132f338441f2bd8741dc58636682" candidate_uuid: "a808fc21ca284713976e332909983ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5836da58aa94012b837968cb1c58a23" is_pre_election: true
I20260619 02:28:37.539841 17685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290f132f338441f2bd8741dc58636682" candidate_uuid: "a808fc21ca284713976e332909983ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d349ba42f73848e2afddb406b27b4ee0" is_pre_election: true
I20260619 02:28:37.540129 17551 raft_consensus.cc:2454] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a808fc21ca284713976e332909983ad9 in term 0.
I20260619 02:28:37.540146 17685 raft_consensus.cc:2454] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a808fc21ca284713976e332909983ad9 in term 0.
I20260619 02:28:37.540580 17755 leader_election.cc:302] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a808fc21ca284713976e332909983ad9, f5836da58aa94012b837968cb1c58a23; no voters:
I20260619 02:28:37.540894 17889 raft_consensus.cc:2781] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:28:37.540980 17889 raft_consensus.cc:482] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:28:37.541024 17889 raft_consensus.cc:3037] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:37.542079 17889 raft_consensus.cc:504] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5836da58aa94012b837968cb1c58a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34757 } } peers { permanent_uuid: "a808fc21ca284713976e332909983ad9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } } peers { permanent_uuid: "d349ba42f73848e2afddb406b27b4ee0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33649 } }
I20260619 02:28:37.542402 17889 leader_election.cc:288] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [CANDIDATE]: Term 1 election: Requested vote from peers f5836da58aa94012b837968cb1c58a23 (127.0.98.1:34757), d349ba42f73848e2afddb406b27b4ee0 (127.0.98.2:33649)
I20260619 02:28:37.542696 17551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290f132f338441f2bd8741dc58636682" candidate_uuid: "a808fc21ca284713976e332909983ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5836da58aa94012b837968cb1c58a23"
I20260619 02:28:37.542758 17685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290f132f338441f2bd8741dc58636682" candidate_uuid: "a808fc21ca284713976e332909983ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d349ba42f73848e2afddb406b27b4ee0"
I20260619 02:28:37.542848 17551 raft_consensus.cc:3037] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:37.542872 17685 raft_consensus.cc:3037] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:37.544175 17551 raft_consensus.cc:2454] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a808fc21ca284713976e332909983ad9 in term 1.
I20260619 02:28:37.544312 17685 raft_consensus.cc:2454] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a808fc21ca284713976e332909983ad9 in term 1.
I20260619 02:28:37.544466 17755 leader_election.cc:302] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a808fc21ca284713976e332909983ad9, f5836da58aa94012b837968cb1c58a23; no voters:
I20260619 02:28:37.544723 17889 raft_consensus.cc:2781] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:37.544917 17889 raft_consensus.cc:686] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [term 1 LEADER]: Becoming Leader. State: Replica: a808fc21ca284713976e332909983ad9, State: Running, Role: LEADER
I20260619 02:28:37.545261 17889 consensus_queue.cc:237] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5836da58aa94012b837968cb1c58a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34757 } } peers { permanent_uuid: "a808fc21ca284713976e332909983ad9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } } peers { permanent_uuid: "d349ba42f73848e2afddb406b27b4ee0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33649 } }
I20260619 02:28:37.548238 17404 catalog_manager.cc:5495] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a808fc21ca284713976e332909983ad9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a808fc21ca284713976e332909983ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5836da58aa94012b837968cb1c58a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a808fc21ca284713976e332909983ad9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d349ba42f73848e2afddb406b27b4ee0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20260619 02:28:37.589478 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.586319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.0.98.3:36681)
0619 02:28:37.586451 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:37.586455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:37.586468 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:28:37.586596 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:37.586599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:37.586659 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:37.586748 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:28:37.586958 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.587442 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.588226 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.588427 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.588501 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.588737 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:28:37.589173 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:28:37.589265 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:37.589310 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 02:28:37.589356 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:28:37.595186 17551 raft_consensus.cc:1261] T 290f132f338441f2bd8741dc58636682 P f5836da58aa94012b837968cb1c58a23 [term 1 FOLLOWER]: Refusing update from remote peer a808fc21ca284713976e332909983ad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:28:37.595186 17685 raft_consensus.cc:1261] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0 [term 1 FOLLOWER]: Refusing update from remote peer a808fc21ca284713976e332909983ad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:28:37.595917 17892 consensus_queue.cc:1035] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5836da58aa94012b837968cb1c58a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:28:37.596093 17889 consensus_queue.cc:1035] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d349ba42f73848e2afddb406b27b4ee0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:28:37.603219 17895 mvcc.cc:205] Tried to move back new op lower bound from 7298400737662271488 to 7298400737467899904. Current Snapshot: MvccSnapshot[applied={T|T < 7298400737662271488}]
I20260619 02:28:37.604467 17896 mvcc.cc:205] Tried to move back new op lower bound from 7298400737662271488 to 7298400737467899904. Current Snapshot: MvccSnapshot[applied={T|T < 7298400737662271488}]
I20260619 02:28:37.611526 17406 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:28:37.611712 17406 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:28:37.614111 17406 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 376cd352135a45439c4d3d4584ce5579: Sending DeleteTablet for 3 replicas of tablet 290f132f338441f2bd8741dc58636682
I20260619 02:28:37.614823 17531 tablet_service.cc:1514] Processing DeleteTablet for tablet 290f132f338441f2bd8741dc58636682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55408
I20260619 02:28:37.614959 17799 tablet_service.cc:1514] Processing DeleteTablet for tablet 290f132f338441f2bd8741dc58636682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60826
I20260619 02:28:37.615067 17665 tablet_service.cc:1514] Processing DeleteTablet for tablet 290f132f338441f2bd8741dc58636682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37872
I20260619 02:28:37.615486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 17468
I20260619 02:28:37.615514 17907 tablet_replica.cc:330] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0: stopping tablet replica
I20260619 02:28:37.615888 17907 raft_consensus.cc:2229] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:28:37.616237 17907 raft_consensus.cc:2258] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:28:37.617761 17906 tablet_replica.cc:330] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9: stopping tablet replica
I20260619 02:28:37.618009 17907 ts_tablet_manager.cc:1905] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:28:37.618006 17906 raft_consensus.cc:2229] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:28:37.618355 17906 raft_consensus.cc:2258] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:28:37.619552 17906 ts_tablet_manager.cc:1905] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:28:37.621109 17907 ts_tablet_manager.cc:1918] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:28:37.621256 17907 log.cc:1196] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/290f132f338441f2bd8741dc58636682
I20260619 02:28:37.621665 17907 ts_tablet_manager.cc:1939] T 290f132f338441f2bd8741dc58636682 P d349ba42f73848e2afddb406b27b4ee0: Deleting consensus metadata
I20260619 02:28:37.622351 17906 ts_tablet_manager.cc:1918] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:28:37.622450 17906 log.cc:1196] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/290f132f338441f2bd8741dc58636682
I20260619 02:28:37.622753 17906 ts_tablet_manager.cc:1939] T 290f132f338441f2bd8741dc58636682 P a808fc21ca284713976e332909983ad9: Deleting consensus metadata
I20260619 02:28:37.623114 17390 catalog_manager.cc:4841] TS d349ba42f73848e2afddb406b27b4ee0 (127.0.98.2:33649): tablet 290f132f338441f2bd8741dc58636682 (table test-table [id=5a5a277ba12b4a589239e36cf452271a]) successfully deleted
I20260619 02:28:37.623752 17390 catalog_manager.cc:4841] TS a808fc21ca284713976e332909983ad9 (127.0.98.3:36681): tablet 290f132f338441f2bd8741dc58636682 (table test-table [id=5a5a277ba12b4a589239e36cf452271a]) successfully deleted
W20260619 02:28:37.626078 17393 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34757 (error 108)
I20260619 02:28:37.626837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 17602
I20260619 02:28:37.626984 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.626690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34757 (local address 127.0.0.1:55414)
0619 02:28:37.626810 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:37.626876 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 02:28:37.627592 17393 catalog_manager.cc:4584] TS f5836da58aa94012b837968cb1c58a23 (127.0.98.1:34757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 290f132f338441f2bd8741dc58636682: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34757: connect: Connection refused (error 111)
I20260619 02:28:37.635640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 17736
I20260619 02:28:37.642009 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 17374
2026-06-19T02:28:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2628 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T02:28:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:37Z Disabled control of system clock
I20260619 02:28:37.671082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37993
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:28:37.776050 17912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:37.776309 17912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:37.776343 17912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:37.780241 17912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:37.780293 17912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:37.780308 17912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:37.780325 17912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:37.780340 17912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:37.784963 17912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37993
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:37.786052 17912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:37.787390 17912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:37.795492 17920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:37.795531 17918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:37.795526 17917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:37.795718 17912 server_base.cc:1034] running on GCE node
I20260619 02:28:37.796311 17912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:37.797362 17912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:37.798537 17912 hybrid_clock.cc:648] HybridClock initialized: now 1781836117798525 us; error 54 us; skew 500 ppm
I20260619 02:28:37.800261 17912 webserver.cc:460] Webserver started at http://127.0.98.62:34857/ using document root <none> and password file <none>
I20260619 02:28:37.800802 17912 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:37.800885 17912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:37.801081 17912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:37.803056 17912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "2cf2a95522664508b52eea09dfcdc867"
format_stamp: "Formatted at 2026-06-19 02:28:37 on dist-test-slave-xkrt"
server_key: "5e04786293e2f6d2237780527d45f7eb"
server_key_iv: "3af421789193da82421e3a2e5f8982ad"
server_key_version: "encryptionkey@0"
I20260619 02:28:37.803610 17912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "2cf2a95522664508b52eea09dfcdc867"
format_stamp: "Formatted at 2026-06-19 02:28:37 on dist-test-slave-xkrt"
server_key: "5e04786293e2f6d2237780527d45f7eb"
server_key_iv: "3af421789193da82421e3a2e5f8982ad"
server_key_version: "encryptionkey@0"
I20260619 02:28:37.807106 17912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 02:28:37.809346 17926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:37.810619 17912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:28:37.810753 17912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "2cf2a95522664508b52eea09dfcdc867"
format_stamp: "Formatted at 2026-06-19 02:28:37 on dist-test-slave-xkrt"
server_key: "5e04786293e2f6d2237780527d45f7eb"
server_key_iv: "3af421789193da82421e3a2e5f8982ad"
server_key_version: "encryptionkey@0"
I20260619 02:28:37.810856 17912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:37.820386 17912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:37.820988 17912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:37.821173 17912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:37.827821 17912 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37993
I20260619 02:28:37.827828 17978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37993 every 8 connection(s)
I20260619 02:28:37.828723 17912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:37.832108 17979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:37.836004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 17912
I20260619 02:28:37.836146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:37.836449 392 external_mini_cluster.cc:1428] Setting key 742e5248b9c8dcf8095daa78576fddc1
I20260619 02:28:37.838101 17979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867: Bootstrap starting.
I20260619 02:28:37.840382 17979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:37.841099 17979 log.cc:824] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:37.843166 17979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867: No bootstrap required, opened a new log
I20260619 02:28:37.845270 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:37.837846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.0.98.62:37993)
0619 02:28:37.838446 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:37.838460 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:37.838494 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:28:37.839099 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:37.839114 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:37.839491 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:37.839779 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:37.840101 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.841103 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.842061 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:37.842719 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:37.842889 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:37.843646 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:37.843651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:37.843662 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:37.844043 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:37.844097 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:37.844116 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:37.844190 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:37.844352 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 02:28:37.844618 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":131,"threads_started":1}
I20260619 02:28:37.846897 17979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf2a95522664508b52eea09dfcdc867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260619 02:28:37.847136 17979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:37.847203 17979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf2a95522664508b52eea09dfcdc867, State: Initialized, Role: FOLLOWER
I20260619 02:28:37.847653 17979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf2a95522664508b52eea09dfcdc867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260619 02:28:37.847800 17979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:37.847887 17979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:37.847985 17979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:37.848836 17979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf2a95522664508b52eea09dfcdc867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260619 02:28:37.849164 17979 leader_election.cc:302] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cf2a95522664508b52eea09dfcdc867; no voters:
I20260619 02:28:37.849453 17979 leader_election.cc:288] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:37.849620 17984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:37.849854 17984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [term 1 LEADER]: Becoming Leader. State: Replica: 2cf2a95522664508b52eea09dfcdc867, State: Running, Role: LEADER
I20260619 02:28:37.850137 17984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf2a95522664508b52eea09dfcdc867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260619 02:28:37.850667 17979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:37.851950 17986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cf2a95522664508b52eea09dfcdc867. Latest consensus state: current_term: 1 leader_uuid: "2cf2a95522664508b52eea09dfcdc867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf2a95522664508b52eea09dfcdc867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } }
I20260619 02:28:37.852064 17986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:37.852288 17985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cf2a95522664508b52eea09dfcdc867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf2a95522664508b52eea09dfcdc867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } }
I20260619 02:28:37.852380 17985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:37.853438 17993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:37.856148 17993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:37.861320 17993 catalog_manager.cc:1347] Generated new cluster ID: e06f032beb954937b42cad56bb564ddf
I20260619 02:28:37.861383 17993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:37.871335 17993 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:37.872437 17993 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:37.879848 17993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867: Generated new TSK 0
I20260619 02:28:37.880525 17993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:28:37.888830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--builtin_ntp_servers=127.0.98.20:45841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:28:37.997867 18003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:37.998107 18003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:37.998145 18003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:38.002140 18003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:38.002187 18003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:38.002264 18003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:28:38.006992 18003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:38.008018 18003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:38.009331 18003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:38.018847 18011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:38.018901 18003 server_base.cc:1034] running on GCE node
W20260619 02:28:38.018839 18009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:38.018862 18008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:38.019438 18003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:38.020037 18003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:38.021219 18003 hybrid_clock.cc:648] HybridClock initialized: now 1781836118021205 us; error 25 us; skew 500 ppm
I20260619 02:28:38.023002 18003 webserver.cc:460] Webserver started at http://127.0.98.1:39149/ using document root <none> and password file <none>
I20260619 02:28:38.023545 18003 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:38.023633 18003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:38.023861 18003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:38.025861 18003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "8090d7e8e4c04f8580b584ea9f4b7c43"
format_stamp: "Formatted at 2026-06-19 02:28:38 on dist-test-slave-xkrt"
server_key: "922658116979857a6736055d23b0f291"
server_key_iv: "e85b5c5a62aa1051b81a9893918c65fd"
server_key_version: "encryptionkey@0"
I20260619 02:28:38.026399 18003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8090d7e8e4c04f8580b584ea9f4b7c43"
format_stamp: "Formatted at 2026-06-19 02:28:38 on dist-test-slave-xkrt"
server_key: "922658116979857a6736055d23b0f291"
server_key_iv: "e85b5c5a62aa1051b81a9893918c65fd"
server_key_version: "encryptionkey@0"
I20260619 02:28:38.029866 18003 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:28:38.032133 18017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:38.033239 18003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:28:38.033366 18003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "8090d7e8e4c04f8580b584ea9f4b7c43"
format_stamp: "Formatted at 2026-06-19 02:28:38 on dist-test-slave-xkrt"
server_key: "922658116979857a6736055d23b0f291"
server_key_iv: "e85b5c5a62aa1051b81a9893918c65fd"
server_key_version: "encryptionkey@0"
I20260619 02:28:38.033473 18003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:38.046909 18003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:38.047519 18003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:38.047719 18003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:38.048255 18003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:38.049132 18003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:38.049202 18003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:38.049263 18003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:38.049315 18003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:38.058054 18003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34107
I20260619 02:28:38.058084 18130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34107 every 8 connection(s)
I20260619 02:28:38.058917 18003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:28:38.064507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 18003
I20260619 02:28:38.064649 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:28:38.064980 392 external_mini_cluster.cc:1428] Setting key b80c723b4353af504d1c2f77099ad8bb
I20260619 02:28:38.067004 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--builtin_ntp_servers=127.0.98.20:45841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:28:38.068393 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.060641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39609 (local address 127.0.98.62:37993)
0619 02:28:38.060807 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:38.060812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:38.061853 (+ 1041us) server_negotiation.cc:407] Connection header received
0619 02:28:38.063184 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:38.063187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:38.063238 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:38.063312 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:38.064187 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.064831 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.066295 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.066585 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.066664 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.067878 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:38.067881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:38.067883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:38.067895 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:38.067913 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:38.067921 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:38.067977 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:38.068166 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 02:28:38.068224 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:28:38.069193 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.061028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.1:39609)
0619 02:28:38.061707 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:38.061748 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:38.062972 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:38.063475 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:38.063482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:38.063562 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:38.064040 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.064052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.064958 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.064962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.066172 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.066185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.066739 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.066744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.066958 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.067582 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:28:38.067587 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:38.067643 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:28:38.067724 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:38.068004 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:38.068015 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:38.068024 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:28:38.068158 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 02:28:38.068464 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
I20260619 02:28:38.070359 18131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
I20260619 02:28:38.070590 18131 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:38.071211 18131 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
I20260619 02:28:38.072882 17943 ts_manager.cc:194] Registered new tserver with Master: 8090d7e8e4c04f8580b584ea9f4b7c43 (127.0.98.1:34107)
I20260619 02:28:38.074086 17943 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39609
W20260619 02:28:38.178804 18134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:38.179160 18134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:38.179242 18134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:38.183264 18134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:38.183346 18134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:38.183434 18134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:28:38.188126 18134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:38.189232 18134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:38.190636 18134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:38.200423 18140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:38.200498 18139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:38.200503 18142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:38.200552 18134 server_base.cc:1034] running on GCE node
I20260619 02:28:38.201033 18134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:38.201678 18134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:38.202840 18134 hybrid_clock.cc:648] HybridClock initialized: now 1781836118202824 us; error 30 us; skew 500 ppm
I20260619 02:28:38.204617 18134 webserver.cc:460] Webserver started at http://127.0.98.2:43103/ using document root <none> and password file <none>
I20260619 02:28:38.205189 18134 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:38.205282 18134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:38.205508 18134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:38.207515 18134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "48425e6087194a8b91f39eb5916ee6ce"
format_stamp: "Formatted at 2026-06-19 02:28:38 on dist-test-slave-xkrt"
server_key: "83d29d9cd897b2a239ca75ac72b91a67"
server_key_iv: "eb060871a52cee04b4193ba23d3d10ff"
server_key_version: "encryptionkey@0"
I20260619 02:28:38.208057 18134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "48425e6087194a8b91f39eb5916ee6ce"
format_stamp: "Formatted at 2026-06-19 02:28:38 on dist-test-slave-xkrt"
server_key: "83d29d9cd897b2a239ca75ac72b91a67"
server_key_iv: "eb060871a52cee04b4193ba23d3d10ff"
server_key_version: "encryptionkey@0"
I20260619 02:28:38.211495 18134 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 02:28:38.213886 18148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:38.215209 18134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:28:38.215349 18134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "48425e6087194a8b91f39eb5916ee6ce"
format_stamp: "Formatted at 2026-06-19 02:28:38 on dist-test-slave-xkrt"
server_key: "83d29d9cd897b2a239ca75ac72b91a67"
server_key_iv: "eb060871a52cee04b4193ba23d3d10ff"
server_key_version: "encryptionkey@0"
I20260619 02:28:38.215459 18134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:38.247720 18134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:38.248430 18134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:38.248641 18134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:38.249241 18134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:38.250195 18134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:38.250274 18134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:38.250345 18134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:38.250396 18134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:38.259686 18134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35431
I20260619 02:28:38.259712 18261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35431 every 8 connection(s)
I20260619 02:28:38.260566 18134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:28:38.263954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 18134
I20260619 02:28:38.264098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:28:38.264405 392 external_mini_cluster.cc:1428] Setting key a9f8b7b6f2bd988813e05f865893304d
I20260619 02:28:38.266625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--builtin_ntp_servers=127.0.98.20:45841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:28:38.270833 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.262451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35139 (local address 127.0.98.62:37993)
0619 02:28:38.262597 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:38.262602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:38.263678 (+ 1076us) server_negotiation.cc:407] Connection header received
0619 02:28:38.265495 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:38.265499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:38.265564 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:38.265652 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:38.266905 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.267423 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.268755 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.268990 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.269107 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.270318 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:38.270322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:38.270326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:38.270345 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:38.270364 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:38.270372 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:38.270447 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:38.270622 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 02:28:38.270688 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:28:38.271668 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.262864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.2:35139)
0619 02:28:38.263513 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:38.263553 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:38.265290 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:38.265877 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:38.265884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:38.265998 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:38.266702 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.266725 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.267562 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.267569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.268625 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.268635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.269154 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.269158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.269326 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.269982 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:28:38.269983 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:38.270046 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:28:38.270167 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:38.270461 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:38.270470 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:38.270476 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:28:38.270605 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 02:28:38.270878 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":102,"threads_started":1}
I20260619 02:28:38.272994 18262 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
I20260619 02:28:38.273273 18262 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:38.273834 18262 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
I20260619 02:28:38.274919 17943 ts_manager.cc:194] Registered new tserver with Master: 48425e6087194a8b91f39eb5916ee6ce (127.0.98.2:35431)
I20260619 02:28:38.275496 17943 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35139
W20260619 02:28:38.374832 18265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:38.375116 18265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:38.375185 18265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:38.379154 18265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:38.379236 18265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:38.379320 18265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:28:38.384135 18265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:38.385138 18265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:38.386516 18265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:38.395756 18270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:38.395793 18265 server_base.cc:1034] running on GCE node
W20260619 02:28:38.395808 18271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:38.395754 18273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:38.396461 18265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:38.397032 18265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:38.398208 18265 hybrid_clock.cc:648] HybridClock initialized: now 1781836118398180 us; error 41 us; skew 500 ppm
I20260619 02:28:38.399863 18265 webserver.cc:460] Webserver started at http://127.0.98.3:46291/ using document root <none> and password file <none>
I20260619 02:28:38.400398 18265 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:38.400483 18265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:38.400712 18265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:38.402750 18265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "8f4752c7abc645ef96562dbb1196955f"
format_stamp: "Formatted at 2026-06-19 02:28:38 on dist-test-slave-xkrt"
server_key: "0f05c3fe495135b8efb830944c8726a2"
server_key_iv: "083ef308ca1959aa6a2baddbfa0d60f2"
server_key_version: "encryptionkey@0"
I20260619 02:28:38.403294 18265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8f4752c7abc645ef96562dbb1196955f"
format_stamp: "Formatted at 2026-06-19 02:28:38 on dist-test-slave-xkrt"
server_key: "0f05c3fe495135b8efb830944c8726a2"
server_key_iv: "083ef308ca1959aa6a2baddbfa0d60f2"
server_key_version: "encryptionkey@0"
I20260619 02:28:38.406730 18265 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:28:38.409005 18279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:38.410166 18265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:28:38.410295 18265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "8f4752c7abc645ef96562dbb1196955f"
format_stamp: "Formatted at 2026-06-19 02:28:38 on dist-test-slave-xkrt"
server_key: "0f05c3fe495135b8efb830944c8726a2"
server_key_iv: "083ef308ca1959aa6a2baddbfa0d60f2"
server_key_version: "encryptionkey@0"
I20260619 02:28:38.410396 18265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:38.429944 18265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:38.430564 18265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:38.430751 18265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:38.431288 18265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:38.432152 18265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:38.432224 18265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:38.432286 18265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:38.432339 18265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:38.440876 18265 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42489
I20260619 02:28:38.440918 18392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42489 every 8 connection(s)
I20260619 02:28:38.441772 18265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:28:38.442613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 18265
I20260619 02:28:38.442734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:28:38.442998 392 external_mini_cluster.cc:1428] Setting key 252fe9d4637b1f92c5921abe66ad0c88
I20260619 02:28:38.450706 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.443653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59381 (local address 127.0.98.62:37993)
0619 02:28:38.443789 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:38.443793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:38.445001 (+ 1208us) server_negotiation.cc:407] Connection header received
0619 02:28:38.446378 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:38.446380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:38.446425 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:38.446495 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:38.447291 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.447819 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.448902 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.449132 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.449216 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.450145 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:38.450148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:38.450150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:38.450205 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:38.450228 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:38.450246 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:38.450303 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:38.450545 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 02:28:38.450597 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:28:38.451562 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.444133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.3:59381)
0619 02:28:38.444851 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:38.444898 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:38.446195 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:38.446612 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:38.446617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:38.446701 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:38.447149 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.447160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.447934 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.447937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.448779 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.448787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.449256 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.449258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.449386 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.449916 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:28:38.449917 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:38.449954 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:28:38.450010 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:38.450387 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:38.450394 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:38.450398 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:28:38.450494 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 02:28:38.450768 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":122,"threads_started":1}
I20260619 02:28:38.452679 18393 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
I20260619 02:28:38.452950 18393 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:38.453503 18393 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
I20260619 02:28:38.454514 17943 ts_manager.cc:194] Registered new tserver with Master: 8f4752c7abc645ef96562dbb1196955f (127.0.98.3:42489)
I20260619 02:28:38.455070 17943 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59381
I20260619 02:28:38.456807 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:28:38.462028 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.458675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.0.98.62:37993)
0619 02:28:38.458813 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:38.458817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:38.458979 (+ 162us) server_negotiation.cc:407] Connection header received
0619 02:28:38.459049 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:38.459051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:38.459088 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:38.459152 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:38.459382 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.459867 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.460633 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.460830 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.460904 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.461582 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:38.461586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:38.461589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:38.461622 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:38.461638 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:38.461647 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:38.461704 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:38.461808 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 02:28:38.461851 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:28:38.464702 17943 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:28:38.467635 17943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:28:38.481846 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.475875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42489 (local address 127.0.0.1:45600)
0619 02:28:38.476858 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:38.476871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:38.476956 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:38.477693 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:38.477699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:38.477868 (+ 169us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:38.477978 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.477984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.479164 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.479168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.480282 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.480292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.481251 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.481255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.481357 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.481400 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 02:28:38.481517 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 02:28:38.481677 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":4,"spinlock_wait_cycles":16768,"thread_start_us":77,"threads_started":1}
I20260619 02:28:38.481856 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.476520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.0.98.3:42489)
0619 02:28:38.477152 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:38.477169 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:38.477207 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:28:38.477282 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:38.477287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:38.477486 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:38.477708 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:38.478132 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.478898 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.480404 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.481125 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.481235 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.481423 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:38.481583 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 02:28:38.481685 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":169,"threads_started":1}
I20260619 02:28:38.482038 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.475777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34107 (local address 127.0.0.1:54054)
0619 02:28:38.476578 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:38.476591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:38.476676 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:38.477693 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:38.477699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:38.477760 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:38.478133 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.478140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.479660 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.479663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.480755 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.480764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.481674 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.481677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.481770 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.481792 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:28:38.481839 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 02:28:38.481912 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":234,"spinlock_wait_cycles":63872,"thread_start_us":115,"threads_started":1}
I20260619 02:28:38.482551 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.475802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57802 (local address 127.0.98.2:35431)
0619 02:28:38.477185 (+ 1383us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:38.477195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:38.477214 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:28:38.477282 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:38.477287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:38.477486 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:38.477896 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:38.478052 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.479235 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.480703 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.481449 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.481570 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.481746 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:38.482300 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 02:28:38.482397 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1288,"thread_start_us":85,"threads_started":1}
I20260619 02:28:38.482772 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.476830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54054 (local address 127.0.98.1:34107)
0619 02:28:38.477246 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:38.477254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:38.477274 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:28:38.477356 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:38.477360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:38.477495 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:38.477805 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:38.478260 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.479542 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.480872 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.481534 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.482271 (+ 737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.482447 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:38.482497 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 02:28:38.482586 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":125,"threads_started":1}
I20260619 02:28:38.483192 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.475537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35431 (local address 127.0.0.1:57802)
0619 02:28:38.476391 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:38.476441 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:38.476585 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:38.477707 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:38.477710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:38.477745 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:38.477955 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.477964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.479369 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.479372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.480575 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.480584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.481813 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.481816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.482141 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.482166 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:28:38.482244 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:28:38.483087 (+ 843us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":42,"spinlock_wait_cycles":8320,"thread_start_us":88,"threads_started":1}
I20260619 02:28:38.485214 18327 tablet_service.cc:1467] Processing CreateTablet for tablet 31e5a57a202d40d581c1261cfee857c2 (DEFAULT_TABLE table=test-table [id=144deadcfe7747f4819baeff41488697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:38.485324 18196 tablet_service.cc:1467] Processing CreateTablet for tablet 31e5a57a202d40d581c1261cfee857c2 (DEFAULT_TABLE table=test-table [id=144deadcfe7747f4819baeff41488697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:38.485212 18065 tablet_service.cc:1467] Processing CreateTablet for tablet 31e5a57a202d40d581c1261cfee857c2 (DEFAULT_TABLE table=test-table [id=144deadcfe7747f4819baeff41488697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:38.486310 18065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e5a57a202d40d581c1261cfee857c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:38.486310 18327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e5a57a202d40d581c1261cfee857c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:38.486318 18196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e5a57a202d40d581c1261cfee857c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:38.492337 18410 tablet_bootstrap.cc:492] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce: Bootstrap starting.
I20260619 02:28:38.493637 18411 tablet_bootstrap.cc:492] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43: Bootstrap starting.
I20260619 02:28:38.494796 18410 tablet_bootstrap.cc:654] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:38.495350 18411 tablet_bootstrap.cc:654] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:38.495556 18410 log.cc:824] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:38.496166 18412 tablet_bootstrap.cc:492] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f: Bootstrap starting.
I20260619 02:28:38.496150 18411 log.cc:824] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:38.497699 18410 tablet_bootstrap.cc:492] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce: No bootstrap required, opened a new log
I20260619 02:28:38.497900 18410 ts_tablet_manager.cc:1397] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:28:38.498045 18411 tablet_bootstrap.cc:492] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43: No bootstrap required, opened a new log
I20260619 02:28:38.498227 18411 ts_tablet_manager.cc:1397] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 02:28:38.501103 18411 raft_consensus.cc:348] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48425e6087194a8b91f39eb5916ee6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35431 } } peers { permanent_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "8f4752c7abc645ef96562dbb1196955f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42489 } }
I20260619 02:28:38.501085 18410 raft_consensus.cc:348] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48425e6087194a8b91f39eb5916ee6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35431 } } peers { permanent_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "8f4752c7abc645ef96562dbb1196955f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42489 } }
I20260619 02:28:38.501299 18411 raft_consensus.cc:374] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:38.501327 18410 raft_consensus.cc:374] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:38.501331 18411 raft_consensus.cc:729] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8090d7e8e4c04f8580b584ea9f4b7c43, State: Initialized, Role: FOLLOWER
I20260619 02:28:38.501405 18410 raft_consensus.cc:729] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48425e6087194a8b91f39eb5916ee6ce, State: Initialized, Role: FOLLOWER
I20260619 02:28:38.501806 18411 consensus_queue.cc:260] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48425e6087194a8b91f39eb5916ee6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35431 } } peers { permanent_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "8f4752c7abc645ef96562dbb1196955f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42489 } }
I20260619 02:28:38.501849 18410 consensus_queue.cc:260] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48425e6087194a8b91f39eb5916ee6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35431 } } peers { permanent_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "8f4752c7abc645ef96562dbb1196955f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42489 } }
I20260619 02:28:38.502772 18411 ts_tablet_manager.cc:1428] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20260619 02:28:38.502745 18262 heartbeater.cc:503] Master 127.0.98.62:37993 was elected leader, sending a full tablet report...
I20260619 02:28:38.502756 18410 ts_tablet_manager.cc:1428] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:28:38.503064 18131 heartbeater.cc:503] Master 127.0.98.62:37993 was elected leader, sending a full tablet report...
I20260619 02:28:38.503525 18412 tablet_bootstrap.cc:654] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:38.504386 18412 log.cc:824] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:38.506119 18412 tablet_bootstrap.cc:492] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f: No bootstrap required, opened a new log
I20260619 02:28:38.506299 18412 ts_tablet_manager.cc:1397] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 02:28:38.509670 18412 raft_consensus.cc:348] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48425e6087194a8b91f39eb5916ee6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35431 } } peers { permanent_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "8f4752c7abc645ef96562dbb1196955f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42489 } }
I20260619 02:28:38.509876 18412 raft_consensus.cc:374] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:38.509907 18412 raft_consensus.cc:729] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4752c7abc645ef96562dbb1196955f, State: Initialized, Role: FOLLOWER
I20260619 02:28:38.510232 18412 consensus_queue.cc:260] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48425e6087194a8b91f39eb5916ee6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35431 } } peers { permanent_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "8f4752c7abc645ef96562dbb1196955f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42489 } }
I20260619 02:28:38.510772 18393 heartbeater.cc:503] Master 127.0.98.62:37993 was elected leader, sending a full tablet report...
I20260619 02:28:38.511075 18412 ts_tablet_manager.cc:1428] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
W20260619 02:28:38.511440 18263 tablet.cc:2369] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:28:38.560032 18132 tablet.cc:2369] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:28:38.567984 18415 raft_consensus.cc:482] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:28:38.568222 18415 raft_consensus.cc:504] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48425e6087194a8b91f39eb5916ee6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35431 } } peers { permanent_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "8f4752c7abc645ef96562dbb1196955f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42489 } }
I20260619 02:28:38.569339 18415 leader_election.cc:288] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48425e6087194a8b91f39eb5916ee6ce (127.0.98.2:35431), 8f4752c7abc645ef96562dbb1196955f (127.0.98.3:42489)
I20260619 02:28:38.573107 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.569786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41135 (local address 127.0.98.2:35431)
0619 02:28:38.569906 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:38.569910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:38.569924 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:28:38.569979 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:38.569983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:38.570035 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:38.570115 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:38.570418 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.571058 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.572136 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.572526 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.572644 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.572704 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:38.572897 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 02:28:38.572966 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:28:38.573098 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.569571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35431 (local address 127.0.98.1:41135)
0619 02:28:38.569687 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:38.569706 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:38.569815 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:38.570157 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:38.570160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:38.570174 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:38.570290 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.570298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.571175 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.571181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.572004 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.572013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.572650 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.572654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.572773 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.572791 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 02:28:38.572880 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 02:28:38.572950 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":64}
I20260619 02:28:38.573185 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.569696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33421 (local address 127.0.98.3:42489)
0619 02:28:38.569819 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:38.569823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:38.569998 (+ 175us) server_negotiation.cc:407] Connection header received
0619 02:28:38.570147 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:38.570150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:38.570200 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:38.570274 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:38.570599 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.571165 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.572198 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.572588 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.572715 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.572777 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:38.572962 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 02:28:38.573030 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:28:38.573452 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.569609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42489 (local address 127.0.98.1:33421)
0619 02:28:38.569899 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:38.569912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:38.570020 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:38.570309 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:38.570312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:38.570332 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:38.570463 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.570469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.571280 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.571285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.572072 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:38.572080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.572721 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.572725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:38.572842 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.572859 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:28:38.572947 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 02:28:38.573014 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":24,"thread_start_us":68,"threads_started":1}
I20260619 02:28:38.573980 18216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e5a57a202d40d581c1261cfee857c2" candidate_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48425e6087194a8b91f39eb5916ee6ce" is_pre_election: true
I20260619 02:28:38.574035 18347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e5a57a202d40d581c1261cfee857c2" candidate_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4752c7abc645ef96562dbb1196955f" is_pre_election: true
I20260619 02:28:38.574298 18216 raft_consensus.cc:2454] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8090d7e8e4c04f8580b584ea9f4b7c43 in term 0.
I20260619 02:28:38.574298 18347 raft_consensus.cc:2454] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8090d7e8e4c04f8580b584ea9f4b7c43 in term 0.
I20260619 02:28:38.574767 18021 leader_election.cc:302] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48425e6087194a8b91f39eb5916ee6ce, 8090d7e8e4c04f8580b584ea9f4b7c43; no voters:
I20260619 02:28:38.575007 18415 raft_consensus.cc:2781] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:28:38.575088 18415 raft_consensus.cc:482] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:28:38.575145 18415 raft_consensus.cc:3037] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:38.576117 18415 raft_consensus.cc:504] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48425e6087194a8b91f39eb5916ee6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35431 } } peers { permanent_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "8f4752c7abc645ef96562dbb1196955f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42489 } }
I20260619 02:28:38.576485 18415 leader_election.cc:288] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [CANDIDATE]: Term 1 election: Requested vote from peers 48425e6087194a8b91f39eb5916ee6ce (127.0.98.2:35431), 8f4752c7abc645ef96562dbb1196955f (127.0.98.3:42489)
I20260619 02:28:38.576833 18216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e5a57a202d40d581c1261cfee857c2" candidate_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48425e6087194a8b91f39eb5916ee6ce"
I20260619 02:28:38.576951 18216 raft_consensus.cc:3037] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:38.576962 18347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e5a57a202d40d581c1261cfee857c2" candidate_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4752c7abc645ef96562dbb1196955f"
I20260619 02:28:38.577108 18347 raft_consensus.cc:3037] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:38.577988 18216 raft_consensus.cc:2454] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8090d7e8e4c04f8580b584ea9f4b7c43 in term 1.
I20260619 02:28:38.578305 18021 leader_election.cc:302] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48425e6087194a8b91f39eb5916ee6ce, 8090d7e8e4c04f8580b584ea9f4b7c43; no voters:
I20260619 02:28:38.578389 18347 raft_consensus.cc:2454] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8090d7e8e4c04f8580b584ea9f4b7c43 in term 1.
I20260619 02:28:38.578522 18415 raft_consensus.cc:2781] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:38.578752 18415 raft_consensus.cc:686] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [term 1 LEADER]: Becoming Leader. State: Replica: 8090d7e8e4c04f8580b584ea9f4b7c43, State: Running, Role: LEADER
I20260619 02:28:38.579067 18415 consensus_queue.cc:237] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48425e6087194a8b91f39eb5916ee6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35431 } } peers { permanent_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } } peers { permanent_uuid: "8f4752c7abc645ef96562dbb1196955f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42489 } }
I20260619 02:28:38.582266 17942 catalog_manager.cc:5495] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8090d7e8e4c04f8580b584ea9f4b7c43 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48425e6087194a8b91f39eb5916ee6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8090d7e8e4c04f8580b584ea9f4b7c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f4752c7abc645ef96562dbb1196955f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42489 } health_report { overall_health: UNKNOWN } } }
I20260619 02:28:38.602064 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.598956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.0.98.1:34107)
0619 02:28:38.599075 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:38.599079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:38.599092 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:28:38.599166 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:38.599168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:38.599220 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:38.599308 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:28:38.599506 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.599997 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.600765 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:38.600966 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:38.601041 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:38.601313 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:28:38.601751 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:28:38.601841 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:38.601896 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 02:28:38.601940 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:28:38.607721 18216 raft_consensus.cc:1261] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce [term 1 FOLLOWER]: Refusing update from remote peer 8090d7e8e4c04f8580b584ea9f4b7c43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:28:38.607717 18347 raft_consensus.cc:1261] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f [term 1 FOLLOWER]: Refusing update from remote peer 8090d7e8e4c04f8580b584ea9f4b7c43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:28:38.608603 18420 consensus_queue.cc:1035] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48425e6087194a8b91f39eb5916ee6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:28:38.608809 18415 consensus_queue.cc:1035] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4752c7abc645ef96562dbb1196955f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:28:38.624732 17943 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:28:38.624903 17943 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:28:38.627251 17943 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cf2a95522664508b52eea09dfcdc867: Sending DeleteTablet for 3 replicas of tablet 31e5a57a202d40d581c1261cfee857c2
I20260619 02:28:38.627972 18065 tablet_service.cc:1514] Processing DeleteTablet for tablet 31e5a57a202d40d581c1261cfee857c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:38 UTC) from {username='slave'} at 127.0.0.1:54054
I20260619 02:28:38.628149 18327 tablet_service.cc:1514] Processing DeleteTablet for tablet 31e5a57a202d40d581c1261cfee857c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:38 UTC) from {username='slave'} at 127.0.0.1:45600
I20260619 02:28:38.628450 18433 tablet_replica.cc:330] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43: stopping tablet replica
I20260619 02:28:38.628443 18196 tablet_service.cc:1514] Processing DeleteTablet for tablet 31e5a57a202d40d581c1261cfee857c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:38 UTC) from {username='slave'} at 127.0.0.1:57802
I20260619 02:28:38.628752 18433 raft_consensus.cc:2229] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:28:38.628922 18435 tablet_replica.cc:330] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce: stopping tablet replica
I20260619 02:28:38.629154 18433 raft_consensus.cc:2258] T 31e5a57a202d40d581c1261cfee857c2 P 8090d7e8e4c04f8580b584ea9f4b7c43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:28:38.629299 18435 raft_consensus.cc:2229] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:28:38.629377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 18003
I20260619 02:28:38.629690 18435 raft_consensus.cc:2258] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:28:38.631306 18435 ts_tablet_manager.cc:1905] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:28:38.631670 18434 tablet_replica.cc:330] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f: stopping tablet replica
I20260619 02:28:38.632009 18434 raft_consensus.cc:2229] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:28:38.632306 18434 raft_consensus.cc:2258] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:28:38.633647 18434 ts_tablet_manager.cc:1905] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:28:38.634153 18435 ts_tablet_manager.cc:1918] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:28:38.634258 18435 log.cc:1196] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/31e5a57a202d40d581c1261cfee857c2
I20260619 02:28:38.634560 18435 ts_tablet_manager.cc:1939] T 31e5a57a202d40d581c1261cfee857c2 P 48425e6087194a8b91f39eb5916ee6ce: Deleting consensus metadata
I20260619 02:28:38.635815 17930 catalog_manager.cc:4841] TS 48425e6087194a8b91f39eb5916ee6ce (127.0.98.2:35431): tablet 31e5a57a202d40d581c1261cfee857c2 (table test-table [id=144deadcfe7747f4819baeff41488697]) successfully deleted
I20260619 02:28:38.636332 18434 ts_tablet_manager.cc:1918] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:28:38.636420 18434 log.cc:1196] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/31e5a57a202d40d581c1261cfee857c2
W20260619 02:28:38.636523 17929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34107 (error 108)
I20260619 02:28:38.636564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 18134
I20260619 02:28:38.636750 18434 ts_tablet_manager.cc:1939] T 31e5a57a202d40d581c1261cfee857c2 P 8f4752c7abc645ef96562dbb1196955f: Deleting consensus metadata
I20260619 02:28:38.638446 17928 catalog_manager.cc:4841] TS 8f4752c7abc645ef96562dbb1196955f (127.0.98.3:42489): tablet 31e5a57a202d40d581c1261cfee857c2 (table test-table [id=144deadcfe7747f4819baeff41488697]) successfully deleted
I20260619 02:28:38.639688 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:38.639310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34107 (local address 127.0.0.1:54064)
0619 02:28:38.639512 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:38.639587 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260619 02:28:38.639868 17929 catalog_manager.cc:4584] TS 8090d7e8e4c04f8580b584ea9f4b7c43 (127.0.98.1:34107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31e5a57a202d40d581c1261cfee857c2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34107: connect: Connection refused (error 111)
I20260619 02:28:38.642751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 18265
I20260619 02:28:38.648102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 17912
2026-06-19T02:28:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1005 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:28:38 dist-test-slave-xkrt krb5kdc[18439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:28:38 dist-test-slave-xkrt krb5kdc[18439](info): set up 2 sockets
Jun 19 02:28:38 dist-test-slave-xkrt krb5kdc[18439](info): commencing operation
krb5kdc: starting...
W20260619 02:28:39.689287 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:28:39 dist-test-slave-xkrt krb5kdc[18439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:28:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:28:39.804390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36543
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:39.910845 18452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:39.911121 18452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:39.911201 18452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:39.915135 18452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:39.915208 18452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:39.915247 18452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:39.915306 18452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:39.915341 18452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:39.920358 18452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:39.921303 18452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:39.922259 18452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:28:39.930881 18452 server_base.cc:1034] running on GCE node
W20260619 02:28:39.930874 18460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:39.930945 18458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:39.930877 18457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:39.931712 18452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:39.932725 18452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:39.933877 18452 hybrid_clock.cc:648] HybridClock initialized: now 1781836119933859 us; error 31 us; skew 500 ppm
Jun 19 02:28:39 dist-test-slave-xkrt krb5kdc[18439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:39.936447 18452 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:28:39.937206 18452 webserver.cc:460] Webserver started at http://127.0.98.62:40301/ using document root <none> and password file <none>
I20260619 02:28:39.937747 18452 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:39.937808 18452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:39.937956 18452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:39.939868 18452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "c847536e3dcf4273a62532511ed1883d"
format_stamp: "Formatted at 2026-06-19 02:28:39 on dist-test-slave-xkrt"
server_key: "79d261b15f967c3197945f71ffbc9a2b"
server_key_iv: "61b156d0cd49f265c35b7ad35a356016"
server_key_version: "encryptionkey@0"
I20260619 02:28:39.940348 18452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "c847536e3dcf4273a62532511ed1883d"
format_stamp: "Formatted at 2026-06-19 02:28:39 on dist-test-slave-xkrt"
server_key: "79d261b15f967c3197945f71ffbc9a2b"
server_key_iv: "61b156d0cd49f265c35b7ad35a356016"
server_key_version: "encryptionkey@0"
I20260619 02:28:39.943784 18452 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:28:39.945989 18467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:39.947149 18452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:28:39.947266 18452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "c847536e3dcf4273a62532511ed1883d"
format_stamp: "Formatted at 2026-06-19 02:28:39 on dist-test-slave-xkrt"
server_key: "79d261b15f967c3197945f71ffbc9a2b"
server_key_iv: "61b156d0cd49f265c35b7ad35a356016"
server_key_version: "encryptionkey@0"
I20260619 02:28:39.947363 18452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:39.959118 18452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:39.961701 18452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:39.961889 18452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:39.968401 18452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36543
I20260619 02:28:39.968411 18519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36543 every 8 connection(s)
I20260619 02:28:39.969269 18452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:39.969571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 18452
I20260619 02:28:39.969712 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:39.970029 392 external_mini_cluster.cc:1428] Setting key 53f84b9b75bc561bbdbe755bd596b001
I20260619 02:28:39.972311 18520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:39.978560 18520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d: Bootstrap starting.
Jun 19 02:28:39 dist-test-slave-xkrt krb5kdc[18439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:28:39.980705 18520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:39.981400 18520 log.cc:824] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:39.983134 18520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d: No bootstrap required, opened a new log
I20260619 02:28:39.984186 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:39.971397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.0.98.62:36543)
0619 02:28:39.971843 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:39.971870 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:39.971906 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:28:39.972806 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:39.972823 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:39.973197 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:39.973469 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:39.974362 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:39.975357 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:39.976500 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:39.976952 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:39.977122 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:39.979872 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:39.979877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:39.979888 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:39.979913 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:39.981622 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:39.982047 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:39.982053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:39.982065 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:39.982168 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:39.982449 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:39.982452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:39.982454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:39.982807 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:39.983016 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:39.983241 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 02:28:39.983508 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":159,"threads_started":1}
I20260619 02:28:39.986603 18520 raft_consensus.cc:348] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c847536e3dcf4273a62532511ed1883d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36543 } }
I20260619 02:28:39.986827 18520 raft_consensus.cc:374] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:39.986907 18520 raft_consensus.cc:729] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c847536e3dcf4273a62532511ed1883d, State: Initialized, Role: FOLLOWER
I20260619 02:28:39.987393 18520 consensus_queue.cc:260] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c847536e3dcf4273a62532511ed1883d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36543 } }
I20260619 02:28:39.987546 18520 raft_consensus.cc:388] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:39.987610 18520 raft_consensus.cc:482] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:39.987704 18520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:39.988580 18520 raft_consensus.cc:504] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c847536e3dcf4273a62532511ed1883d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36543 } }
I20260619 02:28:39.988921 18520 leader_election.cc:302] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c847536e3dcf4273a62532511ed1883d; no voters:
I20260619 02:28:39.989264 18520 leader_election.cc:288] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:39.989374 18525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:39.989609 18525 raft_consensus.cc:686] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [term 1 LEADER]: Becoming Leader. State: Replica: c847536e3dcf4273a62532511ed1883d, State: Running, Role: LEADER
I20260619 02:28:39.989887 18525 consensus_queue.cc:237] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c847536e3dcf4273a62532511ed1883d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36543 } }
I20260619 02:28:39.990424 18520 sys_catalog.cc:564] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:39.991448 18527 sys_catalog.cc:455] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c847536e3dcf4273a62532511ed1883d. Latest consensus state: current_term: 1 leader_uuid: "c847536e3dcf4273a62532511ed1883d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c847536e3dcf4273a62532511ed1883d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36543 } } }
I20260619 02:28:39.991583 18527 sys_catalog.cc:458] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:39.991905 18526 sys_catalog.cc:455] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c847536e3dcf4273a62532511ed1883d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c847536e3dcf4273a62532511ed1883d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36543 } } }
I20260619 02:28:39.992009 18526 sys_catalog.cc:458] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:39.992091 18534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:39.995049 18534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:40.000098 18534 catalog_manager.cc:1347] Generated new cluster ID: d6a960157b1e47b2b99d60ce23167f2e
I20260619 02:28:40.000176 18534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:40.029188 18534 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:40.030153 18534 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:40.036494 18534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d: Generated new TSK 0
I20260619 02:28:40.037161 18534 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:28:40.072489 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36543
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:40.201673 18546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:40.201969 18546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:40.202054 18546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:40.206099 18546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:40.206192 18546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:40.206278 18546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:28:40.211514 18546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:40.212546 18546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:40.213570 18546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:40.222925 18554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:40.222995 18546 server_base.cc:1034] running on GCE node
W20260619 02:28:40.222908 18551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:40.222927 18552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:40.223513 18546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:40.224120 18546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:40.225314 18546 hybrid_clock.cc:648] HybridClock initialized: now 1781836120225302 us; error 25 us; skew 500 ppm
Jun 19 02:28:40 dist-test-slave-xkrt krb5kdc[18439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:40.231088 18546 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:28:40.231940 18546 webserver.cc:460] Webserver started at http://127.0.98.1:38139/ using document root <none> and password file <none>
I20260619 02:28:40.232442 18546 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:40.232496 18546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:40.232651 18546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:40.234642 18546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "84a4168db2cb47a7bc9bd4fc174b0480"
format_stamp: "Formatted at 2026-06-19 02:28:40 on dist-test-slave-xkrt"
server_key: "1215de4c4a9a73d33d080cb265b1a04d"
server_key_iv: "168f1babfc8373b1b126b62ded210ade"
server_key_version: "encryptionkey@0"
I20260619 02:28:40.235136 18546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "84a4168db2cb47a7bc9bd4fc174b0480"
format_stamp: "Formatted at 2026-06-19 02:28:40 on dist-test-slave-xkrt"
server_key: "1215de4c4a9a73d33d080cb265b1a04d"
server_key_iv: "168f1babfc8373b1b126b62ded210ade"
server_key_version: "encryptionkey@0"
I20260619 02:28:40.238622 18546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:28:40.240953 18561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:40.242214 18546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:28:40.242329 18546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "84a4168db2cb47a7bc9bd4fc174b0480"
format_stamp: "Formatted at 2026-06-19 02:28:40 on dist-test-slave-xkrt"
server_key: "1215de4c4a9a73d33d080cb265b1a04d"
server_key_iv: "168f1babfc8373b1b126b62ded210ade"
server_key_version: "encryptionkey@0"
I20260619 02:28:40.242429 18546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:40.285421 18546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:40.288906 18546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:40.289132 18546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:40.289772 18546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:40.290607 18546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:40.290653 18546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:40.290722 18546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:40.290755 18546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:40.299508 18546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33533
I20260619 02:28:40.299574 18674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33533 every 8 connection(s)
I20260619 02:28:40.300382 18546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:28:40 dist-test-slave-xkrt krb5kdc[18439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:28:40.309626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 18546
I20260619 02:28:40.309777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:28:40.310160 392 external_mini_cluster.cc:1428] Setting key 383ff46660b059f9172226984f9b8a67
I20260619 02:28:40.313071 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:40.302155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50313 (local address 127.0.98.62:36543)
0619 02:28:40.302315 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:40.302320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:40.303117 (+ 797us) server_negotiation.cc:407] Connection header received
0619 02:28:40.303941 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:40.303944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:40.304001 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:40.304079 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:40.305169 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.305707 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.306775 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.307000 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.307096 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:40.309387 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:40.309390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:40.309392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:40.309414 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:40.311098 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:40.311585 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:40.311591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:40.311593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:40.311676 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:40.312017 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:40.312022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:40.312023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:40.312197 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:40.312305 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:40.312777 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 02:28:40.312899 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:28:40.313892 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:40.302529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36543 (local address 127.0.98.1:50313)
0619 02:28:40.302964 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:40.303004 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:40.303754 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:40.304232 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:40.304239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:40.304563 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:40.305024 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:40.305037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.305806 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.305809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:40.306667 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:40.306676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.307134 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.307137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:40.307272 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:40.307847 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:28:40.307862 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:40.309219 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:40.311212 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:40.311218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:40.311231 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:40.311479 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:40.311769 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:40.311774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:40.311776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:40.311920 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:40.312312 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:40.312321 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:40.312552 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 02:28:40.312732 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 02:28:40.313029 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":104,"threads_started":1}
I20260619 02:28:40.315042 18675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36543
I20260619 02:28:40.315304 18675 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:40.315779 18675 heartbeater.cc:511] Master 127.0.98.62:36543 requested a full tablet report, sending...
I20260619 02:28:40.317176 18484 ts_manager.cc:194] Registered new tserver with Master: 84a4168db2cb47a7bc9bd4fc174b0480 (127.0.98.1:33533)
I20260619 02:28:40.318320 18484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50313
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:28:40.336539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36543
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:40.445931 18680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:40.446185 18680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:40.446264 18680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:40.450237 18680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:40.450307 18680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:40.450388 18680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:28:40.455624 18680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:40.456566 18680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:40.457631 18680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:40.466781 18686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:40.466823 18685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:40.466822 18688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:40.467195 18680 server_base.cc:1034] running on GCE node
I20260619 02:28:40.467548 18680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:40.468235 18680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:40.469410 18680 hybrid_clock.cc:648] HybridClock initialized: now 1781836120469390 us; error 35 us; skew 500 ppm
Jun 19 02:28:40 dist-test-slave-xkrt krb5kdc[18439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:40.472036 18680 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:28:40.472849 18680 webserver.cc:460] Webserver started at http://127.0.98.2:45405/ using document root <none> and password file <none>
I20260619 02:28:40.473385 18680 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:40.473483 18680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:40.473730 18680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:40.475642 18680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1"
format_stamp: "Formatted at 2026-06-19 02:28:40 on dist-test-slave-xkrt"
server_key: "567bc73ca2aceff6a58d25f138d09ce9"
server_key_iv: "ba72d99d48f4258364c2cce929c232f4"
server_key_version: "encryptionkey@0"
I20260619 02:28:40.476207 18680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1"
format_stamp: "Formatted at 2026-06-19 02:28:40 on dist-test-slave-xkrt"
server_key: "567bc73ca2aceff6a58d25f138d09ce9"
server_key_iv: "ba72d99d48f4258364c2cce929c232f4"
server_key_version: "encryptionkey@0"
I20260619 02:28:40.479723 18680 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:28:40.482124 18695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:40.483319 18680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:28:40.483467 18680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1"
format_stamp: "Formatted at 2026-06-19 02:28:40 on dist-test-slave-xkrt"
server_key: "567bc73ca2aceff6a58d25f138d09ce9"
server_key_iv: "ba72d99d48f4258364c2cce929c232f4"
server_key_version: "encryptionkey@0"
I20260619 02:28:40.483579 18680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:40.501468 18680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:40.504560 18680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:40.504766 18680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:40.505360 18680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:40.506208 18680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:40.506278 18680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:40.506349 18680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:40.506392 18680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:40.515389 18680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41139
I20260619 02:28:40.515426 18808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41139 every 8 connection(s)
I20260619 02:28:40.516244 18680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:28:40.522326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 18680
I20260619 02:28:40.522468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:28:40.522751 392 external_mini_cluster.cc:1428] Setting key 7c51ed168886c5dc8fa70fdb12fab6c3
Jun 19 02:28:40 dist-test-slave-xkrt krb5kdc[18439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:28:40.530201 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:40.517955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41399 (local address 127.0.98.62:36543)
0619 02:28:40.518116 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:40.518120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:40.519012 (+ 892us) server_negotiation.cc:407] Connection header received
0619 02:28:40.519890 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:40.519893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:40.519947 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:40.520027 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:40.521113 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.521732 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.522993 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.523325 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.523443 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:40.526314 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:40.526317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:40.526320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:40.526343 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:40.527844 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:40.528482 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:40.528486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:40.528488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:40.528566 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:40.528959 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:40.528961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:40.528962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:40.529171 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:40.529249 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:40.529946 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 02:28:40.530052 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:28:40.531277 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:40.518408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36543 (local address 127.0.98.2:41399)
0619 02:28:40.518867 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:40.518908 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:40.519695 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:40.520186 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:40.520194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:40.520524 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:40.520970 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:40.520982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.521851 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.521855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:40.522878 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:40.522887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.523477 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.523481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:40.523654 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:40.524293 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:28:40.524313 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:40.526116 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:40.527963 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:40.527972 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:40.527988 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:40.528374 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:40.528664 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:40.528670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:40.528675 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:40.528858 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:40.529280 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:40.529289 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:40.529645 (+ 356us) client_negotiation.cc:769] Sending connection context
0619 02:28:40.529876 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 02:28:40.530232 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":113,"threads_started":1}
I20260619 02:28:40.532470 18809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36543
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:28:40.532706 18809 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:40.533197 18809 heartbeater.cc:511] Master 127.0.98.62:36543 requested a full tablet report, sending...
I20260619 02:28:40.534258 18484 ts_manager.cc:194] Registered new tserver with Master: a367f0d5cd5e4dd183ba8edc6a8c08c1 (127.0.98.2:41139)
I20260619 02:28:40.534790 18484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:28:40.552199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36543
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:40.659154 18814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:40.659382 18814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:40.659457 18814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:40.663424 18814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:40.663472 18814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:40.663558 18814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:28:40.668602 18814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:40.669576 18814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:40.670518 18814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:40.679656 18822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:40.679664 18819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:40.679677 18820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:40.679975 18814 server_base.cc:1034] running on GCE node
I20260619 02:28:40.680341 18814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:40.680891 18814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:40.682054 18814 hybrid_clock.cc:648] HybridClock initialized: now 1781836120682031 us; error 32 us; skew 500 ppm
Jun 19 02:28:40 dist-test-slave-xkrt krb5kdc[18439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:40.685017 18814 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:28:40.685900 18814 webserver.cc:460] Webserver started at http://127.0.98.3:42785/ using document root <none> and password file <none>
I20260619 02:28:40.686468 18814 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:40.686553 18814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:40.686806 18814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:40.688817 18814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "fb83b3c9a9d543c499ee32afcdc49a33"
format_stamp: "Formatted at 2026-06-19 02:28:40 on dist-test-slave-xkrt"
server_key: "bf0895b5a014c1762c72095e1f88fcd6"
server_key_iv: "ce6535ccd30f5210fefd379d4d934fa5"
server_key_version: "encryptionkey@0"
I20260619 02:28:40.689352 18814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fb83b3c9a9d543c499ee32afcdc49a33"
format_stamp: "Formatted at 2026-06-19 02:28:40 on dist-test-slave-xkrt"
server_key: "bf0895b5a014c1762c72095e1f88fcd6"
server_key_iv: "ce6535ccd30f5210fefd379d4d934fa5"
server_key_version: "encryptionkey@0"
I20260619 02:28:40.692971 18814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:28:40.695245 18829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:40.696370 18814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:28:40.696498 18814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "fb83b3c9a9d543c499ee32afcdc49a33"
format_stamp: "Formatted at 2026-06-19 02:28:40 on dist-test-slave-xkrt"
server_key: "bf0895b5a014c1762c72095e1f88fcd6"
server_key_iv: "ce6535ccd30f5210fefd379d4d934fa5"
server_key_version: "encryptionkey@0"
I20260619 02:28:40.696599 18814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:40.711690 18814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:40.714818 18814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:40.715009 18814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:40.715549 18814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:40.716306 18814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:40.716374 18814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:40.716441 18814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:40.716490 18814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:40.724974 18814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37453
I20260619 02:28:40.725014 18942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37453 every 8 connection(s)
I20260619 02:28:40.725908 18814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:28:40.727589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 18814
I20260619 02:28:40.727735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:28:40.728031 392 external_mini_cluster.cc:1428] Setting key 9522bf9f8a3eeb5c0658237435a2d6fc
Jun 19 02:28:40 dist-test-slave-xkrt krb5kdc[18439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:28:40.738178 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:40.727632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33415 (local address 127.0.98.62:36543)
0619 02:28:40.727788 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:40.727793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:40.728729 (+ 936us) server_negotiation.cc:407] Connection header received
0619 02:28:40.729720 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:40.729722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:40.729764 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:40.729820 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:40.730997 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.731522 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.732567 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.732783 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.732857 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:40.735101 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:40.735104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:40.735106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:40.735127 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:40.736496 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:40.736911 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:40.736913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:40.736915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:40.736980 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:40.737277 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:40.737279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:40.737280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:40.737433 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:40.737504 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:40.737976 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 02:28:40.738065 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:28:40.738961 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:40.728052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36543 (local address 127.0.98.3:33415)
0619 02:28:40.728558 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:40.728612 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:40.729511 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:40.729977 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:40.729985 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:40.730426 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:40.730876 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:40.730887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.731622 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.731625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:40.732469 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:40.732477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.732902 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.732905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:40.733028 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:40.733531 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:28:40.733611 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:40.734962 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:40.736587 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:40.736591 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:40.736600 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:40.736831 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:40.737055 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:40.737057 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:40.737059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:40.737182 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:40.737515 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:40.737520 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:40.737781 (+ 261us) client_negotiation.cc:769] Sending connection context
0619 02:28:40.737929 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 02:28:40.738202 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":115,"threads_started":1}
I20260619 02:28:40.740017 18943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36543
I20260619 02:28:40.740281 18943 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:40.740767 18943 heartbeater.cc:511] Master 127.0.98.62:36543 requested a full tablet report, sending...
I20260619 02:28:40.741860 18483 ts_manager.cc:194] Registered new tserver with Master: fb83b3c9a9d543c499ee32afcdc49a33 (127.0.98.3:37453)
I20260619 02:28:40.742435 18483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33415
I20260619 02:28:40.753471 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:28:40.762393 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:40.755418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.0.98.62:36543)
0619 02:28:40.755534 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:40.755537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:40.755693 (+ 156us) server_negotiation.cc:407] Connection header received
0619 02:28:40.755787 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:40.755789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:40.755826 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:40.755890 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:40.756538 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.757048 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.757873 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.758073 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.758143 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:40.759543 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:40.759545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:40.759547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:40.759568 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:40.760940 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:40.761359 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:40.761361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:40.761362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:40.761428 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:40.761703 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:40.761705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:40.761706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:40.761884 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:40.761993 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:40.762213 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 02:28:40.762283 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:28:40.765093 18483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:28:40.768065 18483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:28:40.781783 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:40.775872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33533 (local address 127.0.0.1:49402)
0619 02:28:40.776826 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:40.776842 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:40.776996 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:40.777740 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:40.777743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:40.777764 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:40.777970 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:40.777976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.778976 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.778980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:40.780132 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:40.780153 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.781230 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.781234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:40.781337 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:40.781391 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 02:28:40.781490 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:28:40.781632 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":68,"thread_start_us":88,"threads_started":1}
I20260619 02:28:40.781951 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:40.776537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.0.98.2:41139)
0619 02:28:40.777215 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:40.777230 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:40.777255 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:28:40.777333 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:40.777341 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:40.777520 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:40.777703 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:40.778077 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.779280 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.780725 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.781328 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.781434 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:40.781600 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:40.781693 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 02:28:40.781777 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":111,"threads_started":1}
I20260619 02:28:40.782418 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:40.776156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.0.98.1:33533)
0619 02:28:40.776680 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:40.776696 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:40.776924 (+ 228us) server_negotiation.cc:407] Connection header received
0619 02:28:40.777157 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:40.777165 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:40.777384 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:40.777505 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:40.778105 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.778847 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.780263 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.781086 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.781201 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:40.781340 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:40.782207 (+ 867us) server_negotiation.cc:299] Negotiation successful
0619 02:28:40.782301 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":97,"threads_started":1}
I20260619 02:28:40.781913 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:40.775773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41139 (local address 127.0.0.1:55596)
0619 02:28:40.776199 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:40.776252 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:40.776373 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:40.777698 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:40.777705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:40.777727 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:40.777946 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:40.777956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.779467 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.779469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:40.780595 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:40.780615 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.781433 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.781437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:40.781560 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:40.781584 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:28:40.781651 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 02:28:40.781714 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":20,"thread_start_us":72,"threads_started":1}
I20260619 02:28:40.783569 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:40.776478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37453 (local address 127.0.0.1:33386)
0619 02:28:40.777137 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:40.777150 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:40.777226 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:40.777855 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:40.777863 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:40.777874 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:40.778002 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:40.778007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.780737 (+ 2730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.780740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:40.781813 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:40.781822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.782671 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.782675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:40.783258 (+ 583us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:40.783281 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:28:40.783355 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 02:28:40.783426 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":42,"thread_start_us":86,"threads_started":1}
I20260619 02:28:40.784132 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:40.776628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.0.98.3:37453)
0619 02:28:40.776986 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:40.776997 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:40.777308 (+ 311us) server_negotiation.cc:407] Connection header received
0619 02:28:40.777383 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:40.777388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:40.777583 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:40.777718 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:40.778169 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.779292 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.781938 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:40.782539 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:40.782656 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:40.782794 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:40.783888 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0619 02:28:40.783975 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":117,"threads_started":1}
I20260619 02:28:40.785501 18743 tablet_service.cc:1467] Processing CreateTablet for tablet cfce6b6b831949c68e0e6190378c2dcc (DEFAULT_TABLE table=test-table [id=0b5b4a3cb5894bf1a172e7be2094cb8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:40.785576 18609 tablet_service.cc:1467] Processing CreateTablet for tablet cfce6b6b831949c68e0e6190378c2dcc (DEFAULT_TABLE table=test-table [id=0b5b4a3cb5894bf1a172e7be2094cb8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:40.786531 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfce6b6b831949c68e0e6190378c2dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:40.786628 18609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfce6b6b831949c68e0e6190378c2dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:40.786547 18877 tablet_service.cc:1467] Processing CreateTablet for tablet cfce6b6b831949c68e0e6190378c2dcc (DEFAULT_TABLE table=test-table [id=0b5b4a3cb5894bf1a172e7be2094cb8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:40.787436 18877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfce6b6b831949c68e0e6190378c2dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:40.792610 18960 tablet_bootstrap.cc:492] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1: Bootstrap starting.
I20260619 02:28:40.794176 18961 tablet_bootstrap.cc:492] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480: Bootstrap starting.
I20260619 02:28:40.794934 18960 tablet_bootstrap.cc:654] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:40.795390 18962 tablet_bootstrap.cc:492] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33: Bootstrap starting.
I20260619 02:28:40.796250 18961 tablet_bootstrap.cc:654] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:40.796240 18960 log.cc:824] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:40.797394 18961 log.cc:824] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:40.797588 18962 tablet_bootstrap.cc:654] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:40.798699 18962 log.cc:824] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:40.799213 18961 tablet_bootstrap.cc:492] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480: No bootstrap required, opened a new log
I20260619 02:28:40.799252 18960 tablet_bootstrap.cc:492] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1: No bootstrap required, opened a new log
I20260619 02:28:40.799409 18961 ts_tablet_manager.cc:1397] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:28:40.799444 18960 ts_tablet_manager.cc:1397] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 02:28:40.800711 18962 tablet_bootstrap.cc:492] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33: No bootstrap required, opened a new log
I20260619 02:28:40.800909 18962 ts_tablet_manager.cc:1397] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:28:40.802521 18961 raft_consensus.cc:348] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "84a4168db2cb47a7bc9bd4fc174b0480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33533 } } peers { permanent_uuid: "fb83b3c9a9d543c499ee32afcdc49a33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260619 02:28:40.802747 18961 raft_consensus.cc:374] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:40.802798 18961 raft_consensus.cc:729] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a4168db2cb47a7bc9bd4fc174b0480, State: Initialized, Role: FOLLOWER
I20260619 02:28:40.803223 18961 consensus_queue.cc:260] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "84a4168db2cb47a7bc9bd4fc174b0480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33533 } } peers { permanent_uuid: "fb83b3c9a9d543c499ee32afcdc49a33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260619 02:28:40.803202 18960 raft_consensus.cc:348] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "84a4168db2cb47a7bc9bd4fc174b0480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33533 } } peers { permanent_uuid: "fb83b3c9a9d543c499ee32afcdc49a33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260619 02:28:40.803452 18960 raft_consensus.cc:374] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:40.803512 18960 raft_consensus.cc:729] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a367f0d5cd5e4dd183ba8edc6a8c08c1, State: Initialized, Role: FOLLOWER
I20260619 02:28:40.803970 18675 heartbeater.cc:503] Master 127.0.98.62:36543 was elected leader, sending a full tablet report...
I20260619 02:28:40.803931 18960 consensus_queue.cc:260] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "84a4168db2cb47a7bc9bd4fc174b0480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33533 } } peers { permanent_uuid: "fb83b3c9a9d543c499ee32afcdc49a33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260619 02:28:40.804196 18961 ts_tablet_manager.cc:1428] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:28:40.804837 18960 ts_tablet_manager.cc:1428] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:28:40.805183 18809 heartbeater.cc:503] Master 127.0.98.62:36543 was elected leader, sending a full tablet report...
I20260619 02:28:40.805406 18962 raft_consensus.cc:348] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "84a4168db2cb47a7bc9bd4fc174b0480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33533 } } peers { permanent_uuid: "fb83b3c9a9d543c499ee32afcdc49a33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260619 02:28:40.805670 18962 raft_consensus.cc:374] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:40.805729 18962 raft_consensus.cc:729] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb83b3c9a9d543c499ee32afcdc49a33, State: Initialized, Role: FOLLOWER
I20260619 02:28:40.806126 18962 consensus_queue.cc:260] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "84a4168db2cb47a7bc9bd4fc174b0480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33533 } } peers { permanent_uuid: "fb83b3c9a9d543c499ee32afcdc49a33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260619 02:28:40.806694 18943 heartbeater.cc:503] Master 127.0.98.62:36543 was elected leader, sending a full tablet report...
I20260619 02:28:40.806972 18962 ts_tablet_manager.cc:1428] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260619 02:28:40.976824 18944 tablet.cc:2369] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:28:41.017336 18810 tablet.cc:2369] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:28:41.051751 18676 tablet.cc:2369] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:28:41.158989 18967 raft_consensus.cc:482] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:28:41.159341 18967 raft_consensus.cc:504] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "84a4168db2cb47a7bc9bd4fc174b0480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33533 } } peers { permanent_uuid: "fb83b3c9a9d543c499ee32afcdc49a33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260619 02:28:41.160521 18967 leader_election.cc:288] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84a4168db2cb47a7bc9bd4fc174b0480 (127.0.98.1:33533), fb83b3c9a9d543c499ee32afcdc49a33 (127.0.98.3:37453)
I20260619 02:28:41.164526 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:41.161068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36861 (local address 127.0.98.3:37453)
0619 02:28:41.161195 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:41.161199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:41.161216 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:28:41.161281 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:41.161284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:41.161344 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:41.161425 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:41.161798 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.162332 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.163512 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.163921 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.164012 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:41.164104 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:41.164323 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 02:28:41.164393 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:28:41.164547 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:41.160778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33533 (local address 127.0.98.2:57365)
0619 02:28:41.161232 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:41.161247 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:41.161369 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:41.161647 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:41.161649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:41.161664 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:41.161785 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:41.161792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.162534 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.162536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:41.163411 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:41.163420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.164112 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.164117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:41.164228 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:41.164245 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:28:41.164336 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 02:28:41.164398 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":92,"thread_start_us":94,"threads_started":1}
I20260619 02:28:41.164544 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:41.160921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57365 (local address 127.0.98.1:33533)
0619 02:28:41.161142 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:41.161148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:41.161358 (+ 210us) server_negotiation.cc:407] Connection header received
0619 02:28:41.161470 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:41.161473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:41.161518 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:41.161641 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:41.161899 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.162428 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.163549 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.163952 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.164079 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:41.164156 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:41.164365 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 02:28:41.164440 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 02:28:41.164552 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:41.160778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37453 (local address 127.0.98.2:36861)
0619 02:28:41.161060 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:41.161075 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:41.161201 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:41.161458 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:41.161461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:41.161482 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:41.161680 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:41.161688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.162433 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.162437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:41.163389 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:41.163398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.164043 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.164050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:41.164191 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:41.164219 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 02:28:41.164280 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 02:28:41.164361 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":79,"thread_start_us":81,"threads_started":1}
I20260619 02:28:41.165141 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfce6b6b831949c68e0e6190378c2dcc" candidate_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84a4168db2cb47a7bc9bd4fc174b0480" is_pre_election: true
I20260619 02:28:41.165221 18897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfce6b6b831949c68e0e6190378c2dcc" candidate_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb83b3c9a9d543c499ee32afcdc49a33" is_pre_election: true
I20260619 02:28:41.165436 18629 raft_consensus.cc:2454] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a367f0d5cd5e4dd183ba8edc6a8c08c1 in term 0.
I20260619 02:28:41.165529 18897 raft_consensus.cc:2454] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a367f0d5cd5e4dd183ba8edc6a8c08c1 in term 0.
I20260619 02:28:41.165985 18697 leader_election.cc:302] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84a4168db2cb47a7bc9bd4fc174b0480, a367f0d5cd5e4dd183ba8edc6a8c08c1; no voters:
I20260619 02:28:41.166232 18967 raft_consensus.cc:2781] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:28:41.166329 18967 raft_consensus.cc:482] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:28:41.166385 18967 raft_consensus.cc:3037] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:41.167387 18967 raft_consensus.cc:504] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "84a4168db2cb47a7bc9bd4fc174b0480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33533 } } peers { permanent_uuid: "fb83b3c9a9d543c499ee32afcdc49a33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260619 02:28:41.167737 18967 leader_election.cc:288] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [CANDIDATE]: Term 1 election: Requested vote from peers 84a4168db2cb47a7bc9bd4fc174b0480 (127.0.98.1:33533), fb83b3c9a9d543c499ee32afcdc49a33 (127.0.98.3:37453)
I20260619 02:28:41.168074 18897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfce6b6b831949c68e0e6190378c2dcc" candidate_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb83b3c9a9d543c499ee32afcdc49a33"
I20260619 02:28:41.168085 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfce6b6b831949c68e0e6190378c2dcc" candidate_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84a4168db2cb47a7bc9bd4fc174b0480"
I20260619 02:28:41.168190 18629 raft_consensus.cc:3037] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:41.168200 18897 raft_consensus.cc:3037] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:41.169199 18629 raft_consensus.cc:2454] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a367f0d5cd5e4dd183ba8edc6a8c08c1 in term 1.
I20260619 02:28:41.169273 18897 raft_consensus.cc:2454] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a367f0d5cd5e4dd183ba8edc6a8c08c1 in term 1.
I20260619 02:28:41.169507 18697 leader_election.cc:302] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84a4168db2cb47a7bc9bd4fc174b0480, a367f0d5cd5e4dd183ba8edc6a8c08c1; no voters:
I20260619 02:28:41.169699 18967 raft_consensus.cc:2781] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:41.169905 18967 raft_consensus.cc:686] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [term 1 LEADER]: Becoming Leader. State: Replica: a367f0d5cd5e4dd183ba8edc6a8c08c1, State: Running, Role: LEADER
I20260619 02:28:41.170233 18967 consensus_queue.cc:237] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } } peers { permanent_uuid: "84a4168db2cb47a7bc9bd4fc174b0480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33533 } } peers { permanent_uuid: "fb83b3c9a9d543c499ee32afcdc49a33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260619 02:28:41.173235 18483 catalog_manager.cc:5495] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a367f0d5cd5e4dd183ba8edc6a8c08c1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a367f0d5cd5e4dd183ba8edc6a8c08c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84a4168db2cb47a7bc9bd4fc174b0480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb83b3c9a9d543c499ee32afcdc49a33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20260619 02:28:41.236105 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:41.232790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55612 (local address 127.0.98.2:41139)
0619 02:28:41.232921 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:41.232924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:41.232937 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:28:41.233003 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:41.233005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:41.233064 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:41.233189 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:28:41.233390 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.233900 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.234753 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.234982 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.235111 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:41.235368 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:28:41.235770 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:28:41.235883 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:41.235949 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 02:28:41.235979 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:28:41.241801 18897 raft_consensus.cc:1261] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33 [term 1 FOLLOWER]: Refusing update from remote peer a367f0d5cd5e4dd183ba8edc6a8c08c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:28:41.241801 18629 raft_consensus.cc:1261] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480 [term 1 FOLLOWER]: Refusing update from remote peer a367f0d5cd5e4dd183ba8edc6a8c08c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:28:41.242477 18967 consensus_queue.cc:1035] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84a4168db2cb47a7bc9bd4fc174b0480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:28:41.242671 18971 consensus_queue.cc:1035] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb83b3c9a9d543c499ee32afcdc49a33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:28:41.249094 18974 mvcc.cc:205] Tried to move back new op lower bound from 7298400752599486464 to 7298400752315822080. Current Snapshot: MvccSnapshot[applied={T|T < 7298400752599486464}]
I20260619 02:28:41.249936 18976 mvcc.cc:205] Tried to move back new op lower bound from 7298400752599486464 to 7298400752315822080. Current Snapshot: MvccSnapshot[applied={T|T < 7298400752599486464}]
I20260619 02:28:41.257665 18482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:28:41.257833 18482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:28:41.259980 18482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c847536e3dcf4273a62532511ed1883d: Sending DeleteTablet for 3 replicas of tablet cfce6b6b831949c68e0e6190378c2dcc
I20260619 02:28:41.260763 18877 tablet_service.cc:1514] Processing DeleteTablet for tablet cfce6b6b831949c68e0e6190378c2dcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33386
I20260619 02:28:41.260774 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet cfce6b6b831949c68e0e6190378c2dcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55596
I20260619 02:28:41.260800 18609 tablet_service.cc:1514] Processing DeleteTablet for tablet cfce6b6b831949c68e0e6190378c2dcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49402
I20260619 02:28:41.261245 18985 tablet_replica.cc:330] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480: stopping tablet replica
I20260619 02:28:41.261345 18986 tablet_replica.cc:330] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1: stopping tablet replica
I20260619 02:28:41.261579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 18546
I20260619 02:28:41.261744 18985 raft_consensus.cc:2229] T cfce6b6b831949c68e0e6190378c2dcc P 84a4168db2cb47a7bc9bd4fc174b0480 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:28:41.262156 18984 tablet_replica.cc:330] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33: stopping tablet replica
I20260619 02:28:41.262501 18984 raft_consensus.cc:2229] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:28:41.262889 18986 raft_consensus.cc:2229] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:28:41.263094 18984 raft_consensus.cc:2258] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:28:41.263279 18986 raft_consensus.cc:2258] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:28:41.264726 18984 ts_tablet_manager.cc:1905] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:28:41.267529 18984 ts_tablet_manager.cc:1918] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:28:41.267678 18984 log.cc:1196] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/cfce6b6b831949c68e0e6190378c2dcc
I20260619 02:28:41.268090 18984 ts_tablet_manager.cc:1939] T cfce6b6b831949c68e0e6190378c2dcc P fb83b3c9a9d543c499ee32afcdc49a33: Deleting consensus metadata
I20260619 02:28:41.269295 18468 catalog_manager.cc:4841] TS fb83b3c9a9d543c499ee32afcdc49a33 (127.0.98.3:37453): tablet cfce6b6b831949c68e0e6190378c2dcc (table test-table [id=0b5b4a3cb5894bf1a172e7be2094cb8e]) successfully deleted
I20260619 02:28:41.269454 18986 ts_tablet_manager.cc:1905] T cfce6b6b831949c68e0e6190378c2dcc P a367f0d5cd5e4dd183ba8edc6a8c08c1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 02:28:41.269878 18469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33533 (error 108)
I20260619 02:28:41.269982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 18680
I20260619 02:28:41.271296 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:41.270994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33533 (local address 127.0.0.1:49408)
0619 02:28:41.271157 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:41.271218 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260619 02:28:41.271541 18469 catalog_manager.cc:4584] TS 84a4168db2cb47a7bc9bd4fc174b0480 (127.0.98.1:33533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cfce6b6b831949c68e0e6190378c2dcc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33533: connect: Connection refused (error 111)
I20260619 02:28:41.275851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 18814
I20260619 02:28:41.276290 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:41.275983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41139 (local address 127.0.0.1:55626)
0619 02:28:41.276152 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:41.276207 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 02:28:41.280987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 18452
2026-06-19T02:28:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2633 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T02:28:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:41Z Disabled control of system clock
I20260619 02:28:41.310384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37931
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:28:41.417577 18991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:41.417815 18991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:41.417848 18991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:41.421692 18991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:41.421734 18991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:41.421751 18991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:41.421765 18991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:41.421780 18991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:41.426374 18991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:41.427350 18991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:41.428580 18991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:41.436681 18999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:41.436748 18991 server_base.cc:1034] running on GCE node
W20260619 02:28:41.436697 18996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:41.436697 18997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:41.437438 18991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:41.438468 18991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:41.439652 18991 hybrid_clock.cc:648] HybridClock initialized: now 1781836121439638 us; error 27 us; skew 500 ppm
I20260619 02:28:41.441468 18991 webserver.cc:460] Webserver started at http://127.0.98.62:40591/ using document root <none> and password file <none>
I20260619 02:28:41.442095 18991 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:41.442196 18991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:41.442396 18991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:41.444329 18991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "770714c73383469d9f21fb5f2fd2dcd2"
format_stamp: "Formatted at 2026-06-19 02:28:41 on dist-test-slave-xkrt"
server_key: "e7e79a0bb9954f9497bae358e056a5a3"
server_key_iv: "b156712bf90d4d21f5cd2a15cddcf199"
server_key_version: "encryptionkey@0"
I20260619 02:28:41.444860 18991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "770714c73383469d9f21fb5f2fd2dcd2"
format_stamp: "Formatted at 2026-06-19 02:28:41 on dist-test-slave-xkrt"
server_key: "e7e79a0bb9954f9497bae358e056a5a3"
server_key_iv: "b156712bf90d4d21f5cd2a15cddcf199"
server_key_version: "encryptionkey@0"
I20260619 02:28:41.448228 18991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:28:41.450420 19005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:41.451532 18991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:28:41.451670 18991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "770714c73383469d9f21fb5f2fd2dcd2"
format_stamp: "Formatted at 2026-06-19 02:28:41 on dist-test-slave-xkrt"
server_key: "e7e79a0bb9954f9497bae358e056a5a3"
server_key_iv: "b156712bf90d4d21f5cd2a15cddcf199"
server_key_version: "encryptionkey@0"
I20260619 02:28:41.451772 18991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:41.470542 18991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:41.471216 18991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:41.471403 18991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:41.477990 18991 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37931
I20260619 02:28:41.478004 19057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37931 every 8 connection(s)
I20260619 02:28:41.478866 18991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:41.481837 19058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:41.485755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 18991
I20260619 02:28:41.485888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:41.486159 392 external_mini_cluster.cc:1428] Setting key cdcdb02193bf65bebd90c972ca7c8f89
I20260619 02:28:41.487392 19058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2: Bootstrap starting.
I20260619 02:28:41.489586 19058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:41.490363 19058 log.cc:824] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:41.492139 19058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2: No bootstrap required, opened a new log
I20260619 02:28:41.494963 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:41.487592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.0.98.62:37931)
0619 02:28:41.488185 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:41.488198 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:41.488231 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:28:41.488840 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:41.488856 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:41.489238 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:41.489503 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:41.489941 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.490933 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.491794 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.492311 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.492482 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:41.493278 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:41.493284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:41.493302 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:41.493732 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:41.493784 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:41.493803 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:41.493897 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:41.494046 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 02:28:41.494287 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":116,"threads_started":1}
I20260619 02:28:41.495675 19058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770714c73383469d9f21fb5f2fd2dcd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37931 } }
I20260619 02:28:41.495915 19058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:41.495981 19058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770714c73383469d9f21fb5f2fd2dcd2, State: Initialized, Role: FOLLOWER
I20260619 02:28:41.496433 19058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770714c73383469d9f21fb5f2fd2dcd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37931 } }
I20260619 02:28:41.496590 19058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:41.496660 19058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:41.496754 19058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:41.497808 19058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770714c73383469d9f21fb5f2fd2dcd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37931 } }
I20260619 02:28:41.498090 19058 leader_election.cc:302] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 770714c73383469d9f21fb5f2fd2dcd2; no voters:
I20260619 02:28:41.498378 19058 leader_election.cc:288] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:41.498473 19063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:41.498737 19063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [term 1 LEADER]: Becoming Leader. State: Replica: 770714c73383469d9f21fb5f2fd2dcd2, State: Running, Role: LEADER
I20260619 02:28:41.499051 19063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770714c73383469d9f21fb5f2fd2dcd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37931 } }
I20260619 02:28:41.499580 19058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:41.500792 19064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "770714c73383469d9f21fb5f2fd2dcd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770714c73383469d9f21fb5f2fd2dcd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37931 } } }
I20260619 02:28:41.500916 19064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:41.501171 19065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 770714c73383469d9f21fb5f2fd2dcd2. Latest consensus state: current_term: 1 leader_uuid: "770714c73383469d9f21fb5f2fd2dcd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770714c73383469d9f21fb5f2fd2dcd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37931 } } }
I20260619 02:28:41.501271 19065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:41.501344 19072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:41.503847 19072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:41.509104 19072 catalog_manager.cc:1347] Generated new cluster ID: bdebf30da60d4bf7be7fe9fc34f9eebd
I20260619 02:28:41.509173 19072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:41.546294 19072 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:41.547535 19072 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:41.561774 19072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2: Generated new TSK 0
I20260619 02:28:41.562570 19072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:28:41.570325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37931
--builtin_ntp_servers=127.0.98.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:28:41.678659 19082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:41.678881 19082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:41.678917 19082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:41.682749 19082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:41.682791 19082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:41.682842 19082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:28:41.687399 19082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:41.688282 19082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:41.689504 19082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:41.698390 19088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:41.698413 19087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:41.698365 19090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:41.698532 19082 server_base.cc:1034] running on GCE node
I20260619 02:28:41.698976 19082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:41.699568 19082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:41.700733 19082 hybrid_clock.cc:648] HybridClock initialized: now 1781836121700704 us; error 41 us; skew 500 ppm
I20260619 02:28:41.702535 19082 webserver.cc:460] Webserver started at http://127.0.98.1:42507/ using document root <none> and password file <none>
I20260619 02:28:41.703065 19082 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:41.703150 19082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:41.703356 19082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:41.705271 19082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "a21475a10e414243a716d6dcf0ca0a87"
format_stamp: "Formatted at 2026-06-19 02:28:41 on dist-test-slave-xkrt"
server_key: "bec9c8d390d0e011dda376baf836347f"
server_key_iv: "b02df48f34a46f712b348fe3cda5616b"
server_key_version: "encryptionkey@0"
I20260619 02:28:41.705833 19082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a21475a10e414243a716d6dcf0ca0a87"
format_stamp: "Formatted at 2026-06-19 02:28:41 on dist-test-slave-xkrt"
server_key: "bec9c8d390d0e011dda376baf836347f"
server_key_iv: "b02df48f34a46f712b348fe3cda5616b"
server_key_version: "encryptionkey@0"
I20260619 02:28:41.709129 19082 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 02:28:41.711350 19096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:41.712440 19082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:28:41.712564 19082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "a21475a10e414243a716d6dcf0ca0a87"
format_stamp: "Formatted at 2026-06-19 02:28:41 on dist-test-slave-xkrt"
server_key: "bec9c8d390d0e011dda376baf836347f"
server_key_iv: "b02df48f34a46f712b348fe3cda5616b"
server_key_version: "encryptionkey@0"
I20260619 02:28:41.712671 19082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:41.736959 19082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:41.737596 19082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:41.737784 19082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:41.738325 19082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:41.739205 19082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:41.739274 19082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:41.739341 19082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:41.739382 19082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:41.748910 19082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43227
I20260619 02:28:41.748935 19209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43227 every 8 connection(s)
I20260619 02:28:41.749806 19082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:28:41.755827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 19082
I20260619 02:28:41.755967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:28:41.756307 392 external_mini_cluster.cc:1428] Setting key 94e3e2f9bafaca3bf7895c90d21c1e55
I20260619 02:28:41.758154 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37931
--builtin_ntp_servers=127.0.98.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:28:41.759167 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:41.751520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49377 (local address 127.0.98.62:37931)
0619 02:28:41.751656 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:41.751660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:41.752571 (+ 911us) server_negotiation.cc:407] Connection header received
0619 02:28:41.753932 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:41.753935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:41.753982 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:41.754047 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:41.754951 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.755470 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.756987 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.757206 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.757286 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:41.758553 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:41.758559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:41.758565 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:41.758583 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:41.758616 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:41.758630 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:41.758714 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:41.758890 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 02:28:41.758959 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:28:41.759955 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:41.751841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37931 (local address 127.0.98.1:49377)
0619 02:28:41.752446 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:41.752484 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:41.753728 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:41.754198 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:41.754204 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:41.754283 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:41.754832 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:41.754846 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.755608 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.755613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:41.756732 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:41.756742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.757356 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.757360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:41.757582 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:41.758243 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:28:41.758258 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:41.758316 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:28:41.758397 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:41.758740 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:41.758749 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:41.758754 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:28:41.758871 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 02:28:41.759109 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":109,"threads_started":1}
I20260619 02:28:41.761168 19210 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37931
I20260619 02:28:41.761394 19210 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:41.761938 19210 heartbeater.cc:511] Master 127.0.98.62:37931 requested a full tablet report, sending...
I20260619 02:28:41.763376 19022 ts_manager.cc:194] Registered new tserver with Master: a21475a10e414243a716d6dcf0ca0a87 (127.0.98.1:43227)
I20260619 02:28:41.764488 19022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49377
W20260619 02:28:41.865152 19213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:41.865372 19213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:41.865408 19213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:41.869366 19213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:41.869421 19213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:41.869503 19213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:28:41.874137 19213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:41.875062 19213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:41.876339 19213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:41.885286 19221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:41.885286 19218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:41.885318 19219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:41.885587 19213 server_base.cc:1034] running on GCE node
I20260619 02:28:41.885924 19213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:41.886475 19213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:41.887635 19213 hybrid_clock.cc:648] HybridClock initialized: now 1781836121887622 us; error 24 us; skew 500 ppm
I20260619 02:28:41.889256 19213 webserver.cc:460] Webserver started at http://127.0.98.2:43631/ using document root <none> and password file <none>
I20260619 02:28:41.889827 19213 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:41.889887 19213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:41.890113 19213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:41.891979 19213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "c9332ad5891d48ecbcb04eacb7d9dd59"
format_stamp: "Formatted at 2026-06-19 02:28:41 on dist-test-slave-xkrt"
server_key: "5032bba54d07551cca60264c17188ed9"
server_key_iv: "516ebeb3caad11cf5e7776e055d29e70"
server_key_version: "encryptionkey@0"
I20260619 02:28:41.892494 19213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c9332ad5891d48ecbcb04eacb7d9dd59"
format_stamp: "Formatted at 2026-06-19 02:28:41 on dist-test-slave-xkrt"
server_key: "5032bba54d07551cca60264c17188ed9"
server_key_iv: "516ebeb3caad11cf5e7776e055d29e70"
server_key_version: "encryptionkey@0"
I20260619 02:28:41.895928 19213 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:28:41.898093 19227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:41.899235 19213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:28:41.899389 19213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "c9332ad5891d48ecbcb04eacb7d9dd59"
format_stamp: "Formatted at 2026-06-19 02:28:41 on dist-test-slave-xkrt"
server_key: "5032bba54d07551cca60264c17188ed9"
server_key_iv: "516ebeb3caad11cf5e7776e055d29e70"
server_key_version: "encryptionkey@0"
I20260619 02:28:41.899498 19213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:41.910826 19213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:41.911393 19213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:41.911566 19213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:41.912074 19213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:41.912935 19213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:41.913002 19213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:41.913065 19213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:41.913110 19213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:41.921677 19213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44235
I20260619 02:28:41.921700 19340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44235 every 8 connection(s)
I20260619 02:28:41.922484 19213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:28:41.924080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 19213
I20260619 02:28:41.924218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:28:41.924499 392 external_mini_cluster.cc:1428] Setting key 7a18918f672d7f36e04a0c663d32a4f3
I20260619 02:28:41.926357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37931
--builtin_ntp_servers=127.0.98.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:28:41.931447 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:41.924101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33441 (local address 127.0.98.62:37931)
0619 02:28:41.924271 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:41.924276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:41.925329 (+ 1053us) server_negotiation.cc:407] Connection header received
0619 02:28:41.926736 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:41.926742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:41.926802 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:41.926890 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:41.927843 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.928368 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.929480 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.929775 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.929883 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:41.930965 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:41.930969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:41.930972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:41.930984 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:41.931007 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:41.931015 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:41.931083 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:41.931253 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 02:28:41.931311 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:28:41.932245 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:41.924512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37931 (local address 127.0.98.2:33441)
0619 02:28:41.925184 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:41.925231 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:41.926533 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:41.927040 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:41.927047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:41.927126 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:41.927694 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:41.927714 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.928490 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.928495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:41.929384 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:41.929392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:41.929923 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:41.929927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:41.930096 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:41.930713 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:28:41.930715 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:41.930753 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:28:41.930811 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:41.931098 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:41.931125 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:41.931135 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 02:28:41.931252 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 02:28:41.931507 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":115,"threads_started":1}
I20260619 02:28:41.933421 19341 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37931
I20260619 02:28:41.933668 19341 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:41.934147 19341 heartbeater.cc:511] Master 127.0.98.62:37931 requested a full tablet report, sending...
I20260619 02:28:41.935138 19022 ts_manager.cc:194] Registered new tserver with Master: c9332ad5891d48ecbcb04eacb7d9dd59 (127.0.98.2:44235)
I20260619 02:28:41.935685 19022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:33441
W20260619 02:28:42.036748 19344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:42.036996 19344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:42.037065 19344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:42.040982 19344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:42.041059 19344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:42.041137 19344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:28:42.045960 19344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:42.046893 19344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:42.048171 19344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:42.057279 19352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:42.057324 19349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:42.057317 19344 server_base.cc:1034] running on GCE node
W20260619 02:28:42.057279 19350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:42.057896 19344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:42.058521 19344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:42.059708 19344 hybrid_clock.cc:648] HybridClock initialized: now 1781836122059696 us; error 28 us; skew 500 ppm
I20260619 02:28:42.061674 19344 webserver.cc:460] Webserver started at http://127.0.98.3:33313/ using document root <none> and password file <none>
I20260619 02:28:42.062276 19344 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:42.062366 19344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:42.062618 19344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:42.064548 19344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "af33c0fb4bcd4d7a91b666d5156c98e6"
format_stamp: "Formatted at 2026-06-19 02:28:42 on dist-test-slave-xkrt"
server_key: "df981bf83db92300f8bb7ccd00f56044"
server_key_iv: "7e3f823131841a947a93f8eca926ef90"
server_key_version: "encryptionkey@0"
I20260619 02:28:42.065074 19344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "af33c0fb4bcd4d7a91b666d5156c98e6"
format_stamp: "Formatted at 2026-06-19 02:28:42 on dist-test-slave-xkrt"
server_key: "df981bf83db92300f8bb7ccd00f56044"
server_key_iv: "7e3f823131841a947a93f8eca926ef90"
server_key_version: "encryptionkey@0"
I20260619 02:28:42.068612 19344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 02:28:42.070979 19358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:42.072099 19344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:28:42.072269 19344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "af33c0fb4bcd4d7a91b666d5156c98e6"
format_stamp: "Formatted at 2026-06-19 02:28:42 on dist-test-slave-xkrt"
server_key: "df981bf83db92300f8bb7ccd00f56044"
server_key_iv: "7e3f823131841a947a93f8eca926ef90"
server_key_version: "encryptionkey@0"
I20260619 02:28:42.072377 19344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:42.082942 19344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:42.083523 19344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:42.083699 19344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:42.084236 19344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:42.085088 19344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:42.085162 19344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:42.085246 19344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:42.085292 19344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:42.093734 19344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35473
I20260619 02:28:42.093760 19471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35473 every 8 connection(s)
I20260619 02:28:42.094568 19344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:28:42.103222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 19344
I20260619 02:28:42.103336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:28:42.103437 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.096268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33503 (local address 127.0.98.62:37931)
0619 02:28:42.096394 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:42.096398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:42.097362 (+ 964us) server_negotiation.cc:407] Connection header received
0619 02:28:42.098722 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:42.098725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:42.098779 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:42.098861 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:42.099681 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.100278 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.101468 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.101819 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.101921 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:42.102905 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:42.102909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:42.102911 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:42.102924 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:42.102949 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:42.102957 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:42.103025 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:42.103220 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 02:28:42.103278 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:28:42.103667 392 external_mini_cluster.cc:1428] Setting key f5b231d21793092ad29156e72adf4a6e
I20260619 02:28:42.104321 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.096646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37931 (local address 127.0.98.3:33503)
0619 02:28:42.097226 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:42.097262 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:42.098530 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:42.099024 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:42.099030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:42.099106 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:42.099548 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:42.099559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.100395 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.100398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:42.101346 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:42.101355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.101950 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.101954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:42.102122 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:42.102660 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:28:42.102661 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:42.102697 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:28:42.102759 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:42.103039 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:42.103048 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:42.103057 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:28:42.103171 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 02:28:42.103420 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
I20260619 02:28:42.105350 19472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37931
I20260619 02:28:42.105620 19472 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:42.106125 19472 heartbeater.cc:511] Master 127.0.98.62:37931 requested a full tablet report, sending...
I20260619 02:28:42.107017 19022 ts_manager.cc:194] Registered new tserver with Master: af33c0fb4bcd4d7a91b666d5156c98e6 (127.0.98.3:35473)
I20260619 02:28:42.107527 19022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33503
I20260619 02:28:42.117322 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:28:42.122509 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.119158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.0.98.62:37931)
0619 02:28:42.119263 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:42.119266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:42.119429 (+ 163us) server_negotiation.cc:407] Connection header received
0619 02:28:42.119514 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:42.119517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:42.119551 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:42.119623 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:42.119859 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.120335 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.121085 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.121281 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.121352 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:42.122137 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:42.122140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:42.122142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:42.122152 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:42.122168 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:42.122175 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:42.122263 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:42.122387 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 02:28:42.122416 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:28:42.125088 19022 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:28:42.127998 19022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:28:42.141808 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.135766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44235 (local address 127.0.0.1:52016)
0619 02:28:42.136513 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:42.136527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:42.136615 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:42.137289 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:42.137295 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:42.137316 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:42.137430 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:42.137437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.138691 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.138694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:42.140291 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:42.140302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.141333 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.141336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:42.141410 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:42.141438 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 02:28:42.141509 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 02:28:42.141653 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":146,"thread_start_us":75,"threads_started":1}
I20260619 02:28:42.141851 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.136335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.0.98.2:44235)
0619 02:28:42.136849 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:42.136860 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:42.136878 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:28:42.136939 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:42.136943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:42.137110 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:42.137598 (+ 488us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:42.137714 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.138600 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.140417 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.141222 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.141337 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:42.141514 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:42.141598 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 02:28:42.141699 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":129,"threads_started":1}
I20260619 02:28:42.142419 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.136424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.0.98.1:43227)
0619 02:28:42.137020 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:42.137032 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:42.137245 (+ 213us) server_negotiation.cc:407] Connection header received
0619 02:28:42.137465 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:42.137470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:42.137711 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:42.137827 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:42.138206 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.139095 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.140452 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.141235 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.141337 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:42.141514 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:42.142222 (+ 708us) server_negotiation.cc:299] Negotiation successful
0619 02:28:42.142305 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":110,"threads_started":1}
I20260619 02:28:42.142920 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.135690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35473 (local address 127.0.0.1:45860)
0619 02:28:42.136415 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:42.136462 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:42.136588 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:42.137387 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:42.137389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:42.137402 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:42.137472 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:42.137479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.138634 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.138637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:42.140291 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:42.140302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.142524 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.142529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:42.142624 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:42.142650 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 02:28:42.142721 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 02:28:42.142784 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":179,"thread_start_us":71,"threads_started":1}
I20260619 02:28:42.142932 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.136014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43227 (local address 127.0.0.1:54598)
0619 02:28:42.137149 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:42.137163 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:42.137314 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:42.137997 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:42.137999 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:42.138016 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:42.138090 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:42.138095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.139204 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.139210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:42.140310 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:42.140319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.141638 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.141641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:42.141718 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:42.141738 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 02:28:42.141808 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:28:42.142823 (+ 1015us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"mutex_wait_us":82,"thread_start_us":82,"threads_started":1}
I20260619 02:28:42.144780 19275 tablet_service.cc:1467] Processing CreateTablet for tablet 95db349be65b451fad15353507dee9c9 (DEFAULT_TABLE table=test-table [id=a868a0fc41e84ac099c9cd495ab21ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:42.145874 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.135954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.0.98.3:35473)
0619 02:28:42.136807 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:42.136818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:42.136840 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:28:42.136916 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:42.136922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:42.137110 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:42.137263 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:42.137586 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.138519 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.141606 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.142379 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.142486 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:42.142647 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:42.145681 (+ 3034us) server_negotiation.cc:299] Negotiation successful
0619 02:28:42.145757 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":143,"threads_started":1}
I20260619 02:28:42.145885 19275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95db349be65b451fad15353507dee9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:42.145924 19144 tablet_service.cc:1467] Processing CreateTablet for tablet 95db349be65b451fad15353507dee9c9 (DEFAULT_TABLE table=test-table [id=a868a0fc41e84ac099c9cd495ab21ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:42.146912 19144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95db349be65b451fad15353507dee9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:42.148319 19406 tablet_service.cc:1467] Processing CreateTablet for tablet 95db349be65b451fad15353507dee9c9 (DEFAULT_TABLE table=test-table [id=a868a0fc41e84ac099c9cd495ab21ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:42.149250 19406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95db349be65b451fad15353507dee9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:42.154239 19489 tablet_bootstrap.cc:492] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87: Bootstrap starting.
I20260619 02:28:42.155725 19490 tablet_bootstrap.cc:492] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59: Bootstrap starting.
I20260619 02:28:42.156037 19491 tablet_bootstrap.cc:492] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6: Bootstrap starting.
I20260619 02:28:42.157150 19489 tablet_bootstrap.cc:654] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:42.157889 19490 tablet_bootstrap.cc:654] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:42.158049 19491 tablet_bootstrap.cc:654] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:42.158082 19489 log.cc:824] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:42.158684 19490 log.cc:824] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:42.158806 19491 log.cc:824] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:42.160135 19489 tablet_bootstrap.cc:492] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87: No bootstrap required, opened a new log
I20260619 02:28:42.160329 19489 ts_tablet_manager.cc:1397] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:28:42.160435 19491 tablet_bootstrap.cc:492] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6: No bootstrap required, opened a new log
I20260619 02:28:42.160626 19491 ts_tablet_manager.cc:1397] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 02:28:42.160641 19490 tablet_bootstrap.cc:492] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59: No bootstrap required, opened a new log
I20260619 02:28:42.160820 19490 ts_tablet_manager.cc:1397] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:28:42.163630 19491 raft_consensus.cc:348] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af33c0fb4bcd4d7a91b666d5156c98e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35473 } } peers { permanent_uuid: "c9332ad5891d48ecbcb04eacb7d9dd59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44235 } } peers { permanent_uuid: "a21475a10e414243a716d6dcf0ca0a87" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43227 } }
I20260619 02:28:42.163928 19491 raft_consensus.cc:374] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:42.164011 19491 raft_consensus.cc:729] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af33c0fb4bcd4d7a91b666d5156c98e6, State: Initialized, Role: FOLLOWER
I20260619 02:28:42.164615 19491 consensus_queue.cc:260] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af33c0fb4bcd4d7a91b666d5156c98e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35473 } } peers { permanent_uuid: "c9332ad5891d48ecbcb04eacb7d9dd59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44235 } } peers { permanent_uuid: "a21475a10e414243a716d6dcf0ca0a87" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43227 } }
I20260619 02:28:42.164896 19489 raft_consensus.cc:348] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af33c0fb4bcd4d7a91b666d5156c98e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35473 } } peers { permanent_uuid: "c9332ad5891d48ecbcb04eacb7d9dd59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44235 } } peers { permanent_uuid: "a21475a10e414243a716d6dcf0ca0a87" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43227 } }
I20260619 02:28:42.164935 19490 raft_consensus.cc:348] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af33c0fb4bcd4d7a91b666d5156c98e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35473 } } peers { permanent_uuid: "c9332ad5891d48ecbcb04eacb7d9dd59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44235 } } peers { permanent_uuid: "a21475a10e414243a716d6dcf0ca0a87" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43227 } }
I20260619 02:28:42.165149 19489 raft_consensus.cc:374] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:42.165149 19490 raft_consensus.cc:374] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:42.165203 19489 raft_consensus.cc:729] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a21475a10e414243a716d6dcf0ca0a87, State: Initialized, Role: FOLLOWER
I20260619 02:28:42.165203 19490 raft_consensus.cc:729] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9332ad5891d48ecbcb04eacb7d9dd59, State: Initialized, Role: FOLLOWER
I20260619 02:28:42.165357 19472 heartbeater.cc:503] Master 127.0.98.62:37931 was elected leader, sending a full tablet report...
I20260619 02:28:42.165628 19491 ts_tablet_manager.cc:1428] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:28:42.165655 19490 consensus_queue.cc:260] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af33c0fb4bcd4d7a91b666d5156c98e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35473 } } peers { permanent_uuid: "c9332ad5891d48ecbcb04eacb7d9dd59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44235 } } peers { permanent_uuid: "a21475a10e414243a716d6dcf0ca0a87" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43227 } }
I20260619 02:28:42.165692 19489 consensus_queue.cc:260] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af33c0fb4bcd4d7a91b666d5156c98e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35473 } } peers { permanent_uuid: "c9332ad5891d48ecbcb04eacb7d9dd59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44235 } } peers { permanent_uuid: "a21475a10e414243a716d6dcf0ca0a87" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43227 } }
I20260619 02:28:42.166621 19489 ts_tablet_manager.cc:1428] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 02:28:42.166613 19490 ts_tablet_manager.cc:1428] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 02:28:42.166805 19341 heartbeater.cc:503] Master 127.0.98.62:37931 was elected leader, sending a full tablet report...
I20260619 02:28:42.166842 19210 heartbeater.cc:503] Master 127.0.98.62:37931 was elected leader, sending a full tablet report...
W20260619 02:28:42.173274 19342 tablet.cc:2369] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:28:42.250908 19211 tablet.cc:2369] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:28:42.322400 19496 raft_consensus.cc:482] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:28:42.322593 19496 raft_consensus.cc:504] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af33c0fb4bcd4d7a91b666d5156c98e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35473 } } peers { permanent_uuid: "c9332ad5891d48ecbcb04eacb7d9dd59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44235 } } peers { permanent_uuid: "a21475a10e414243a716d6dcf0ca0a87" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43227 } }
I20260619 02:28:42.323652 19496 leader_election.cc:288] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af33c0fb4bcd4d7a91b666d5156c98e6 (127.0.98.3:35473), c9332ad5891d48ecbcb04eacb7d9dd59 (127.0.98.2:44235)
I20260619 02:28:42.327440 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.323980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35333 (local address 127.0.98.2:44235)
0619 02:28:42.324124 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:42.324130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:42.324378 (+ 248us) server_negotiation.cc:407] Connection header received
0619 02:28:42.324520 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:42.324523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:42.324575 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:42.324654 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:42.324913 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.325434 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.326518 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.326908 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.327016 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:42.327073 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:42.327252 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 02:28:42.327330 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:28:42.327459 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.323869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44235 (local address 127.0.98.1:35333)
0619 02:28:42.324290 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:42.324301 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:42.324414 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:42.324670 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:42.324673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:42.324688 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:42.324810 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:42.324815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.325553 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.325560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:42.326402 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:42.326410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.327031 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.327034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:42.327135 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:42.327155 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 02:28:42.327236 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:28:42.327302 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260619 02:28:42.327487 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.323915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46343 (local address 127.0.98.3:35473)
0619 02:28:42.324072 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:42.324077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:42.324306 (+ 229us) server_negotiation.cc:407] Connection header received
0619 02:28:42.324460 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:42.324467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:42.324510 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:42.324589 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:42.324862 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.325360 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.326553 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.326954 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.327067 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:42.327121 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:42.327307 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 02:28:42.327372 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:28:42.327507 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.323831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35473 (local address 127.0.98.1:46343)
0619 02:28:42.324225 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:42.324237 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:42.324352 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:42.324602 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:42.324605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:42.324621 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:42.324761 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:42.324767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.325469 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.325472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:42.326435 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:42.326445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.327072 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.327075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:42.327184 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:42.327210 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 02:28:42.327297 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 02:28:42.327366 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":68,"thread_start_us":105,"threads_started":1}
I20260619 02:28:42.328091 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95db349be65b451fad15353507dee9c9" candidate_uuid: "a21475a10e414243a716d6dcf0ca0a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9332ad5891d48ecbcb04eacb7d9dd59" is_pre_election: true
I20260619 02:28:42.328088 19426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95db349be65b451fad15353507dee9c9" candidate_uuid: "a21475a10e414243a716d6dcf0ca0a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af33c0fb4bcd4d7a91b666d5156c98e6" is_pre_election: true
I20260619 02:28:42.328393 19295 raft_consensus.cc:2454] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a21475a10e414243a716d6dcf0ca0a87 in term 0.
I20260619 02:28:42.328424 19426 raft_consensus.cc:2454] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a21475a10e414243a716d6dcf0ca0a87 in term 0.
I20260619 02:28:42.328878 19098 leader_election.cc:302] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a21475a10e414243a716d6dcf0ca0a87, c9332ad5891d48ecbcb04eacb7d9dd59; no voters:
I20260619 02:28:42.329150 19496 raft_consensus.cc:2781] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:28:42.329232 19496 raft_consensus.cc:482] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:28:42.329277 19496 raft_consensus.cc:3037] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:42.330279 19496 raft_consensus.cc:504] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af33c0fb4bcd4d7a91b666d5156c98e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35473 } } peers { permanent_uuid: "c9332ad5891d48ecbcb04eacb7d9dd59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44235 } } peers { permanent_uuid: "a21475a10e414243a716d6dcf0ca0a87" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43227 } }
I20260619 02:28:42.330578 19496 leader_election.cc:288] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [CANDIDATE]: Term 1 election: Requested vote from peers af33c0fb4bcd4d7a91b666d5156c98e6 (127.0.98.3:35473), c9332ad5891d48ecbcb04eacb7d9dd59 (127.0.98.2:44235)
I20260619 02:28:42.330852 19426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95db349be65b451fad15353507dee9c9" candidate_uuid: "a21475a10e414243a716d6dcf0ca0a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af33c0fb4bcd4d7a91b666d5156c98e6"
I20260619 02:28:42.330883 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95db349be65b451fad15353507dee9c9" candidate_uuid: "a21475a10e414243a716d6dcf0ca0a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9332ad5891d48ecbcb04eacb7d9dd59"
I20260619 02:28:42.330991 19426 raft_consensus.cc:3037] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:42.331007 19295 raft_consensus.cc:3037] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:42.332145 19426 raft_consensus.cc:2454] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a21475a10e414243a716d6dcf0ca0a87 in term 1.
I20260619 02:28:42.332255 19295 raft_consensus.cc:2454] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a21475a10e414243a716d6dcf0ca0a87 in term 1.
I20260619 02:28:42.332522 19097 leader_election.cc:302] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a21475a10e414243a716d6dcf0ca0a87, af33c0fb4bcd4d7a91b666d5156c98e6; no voters:
I20260619 02:28:42.332715 19496 raft_consensus.cc:2781] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:42.332899 19496 raft_consensus.cc:686] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [term 1 LEADER]: Becoming Leader. State: Replica: a21475a10e414243a716d6dcf0ca0a87, State: Running, Role: LEADER
I20260619 02:28:42.333209 19496 consensus_queue.cc:237] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af33c0fb4bcd4d7a91b666d5156c98e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35473 } } peers { permanent_uuid: "c9332ad5891d48ecbcb04eacb7d9dd59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44235 } } peers { permanent_uuid: "a21475a10e414243a716d6dcf0ca0a87" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43227 } }
I20260619 02:28:42.336438 19022 catalog_manager.cc:5495] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 reported cstate change: term changed from 0 to 1, leader changed from <none> to a21475a10e414243a716d6dcf0ca0a87 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a21475a10e414243a716d6dcf0ca0a87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af33c0fb4bcd4d7a91b666d5156c98e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9332ad5891d48ecbcb04eacb7d9dd59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a21475a10e414243a716d6dcf0ca0a87" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43227 } health_report { overall_health: HEALTHY } } }
W20260619 02:28:42.345409 19473 tablet.cc:2369] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:28:42.374825 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.371634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.0.98.1:43227)
0619 02:28:42.371753 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:42.371757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:42.371771 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:28:42.371864 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:42.371866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:42.371920 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:42.372003 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:28:42.372210 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.372688 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.373506 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:42.373757 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:42.373836 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:42.374074 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:28:42.374478 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:28:42.374584 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:42.374692 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 02:28:42.374720 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:28:42.380496 19426 raft_consensus.cc:1261] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6 [term 1 FOLLOWER]: Refusing update from remote peer a21475a10e414243a716d6dcf0ca0a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:28:42.380497 19295 raft_consensus.cc:1261] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59 [term 1 FOLLOWER]: Refusing update from remote peer a21475a10e414243a716d6dcf0ca0a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:28:42.381166 19496 consensus_queue.cc:1035] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af33c0fb4bcd4d7a91b666d5156c98e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:28:42.381431 19500 consensus_queue.cc:1035] T 95db349be65b451fad15353507dee9c9 P a21475a10e414243a716d6dcf0ca0a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9332ad5891d48ecbcb04eacb7d9dd59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:28:42.387372 19503 mvcc.cc:205] Tried to move back new op lower bound from 7298400757263765504 to 7298400757079490560. Current Snapshot: MvccSnapshot[applied={T|T < 7298400757263765504}]
I20260619 02:28:42.395553 19022 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:28:42.395733 19022 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:28:42.398077 19022 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 770714c73383469d9f21fb5f2fd2dcd2: Sending DeleteTablet for 3 replicas of tablet 95db349be65b451fad15353507dee9c9
I20260619 02:28:42.398734 19275 tablet_service.cc:1514] Processing DeleteTablet for tablet 95db349be65b451fad15353507dee9c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:42 UTC) from {username='slave'} at 127.0.0.1:52016
I20260619 02:28:42.398928 19144 tablet_service.cc:1514] Processing DeleteTablet for tablet 95db349be65b451fad15353507dee9c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:42 UTC) from {username='slave'} at 127.0.0.1:54598
I20260619 02:28:42.399127 19512 tablet_replica.cc:330] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59: stopping tablet replica
I20260619 02:28:42.399175 19406 tablet_service.cc:1514] Processing DeleteTablet for tablet 95db349be65b451fad15353507dee9c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:42 UTC) from {username='slave'} at 127.0.0.1:45860
I20260619 02:28:42.399410 19512 raft_consensus.cc:2229] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:28:42.399688 19514 tablet_replica.cc:330] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6: stopping tablet replica
I20260619 02:28:42.399755 19512 raft_consensus.cc:2258] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:28:42.400049 19514 raft_consensus.cc:2229] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:28:42.400319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 19082
I20260619 02:28:42.400434 19514 raft_consensus.cc:2258] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:28:42.402010 19514 ts_tablet_manager.cc:1905] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:28:42.403483 19512 ts_tablet_manager.cc:1905] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:28:42.404817 19514 ts_tablet_manager.cc:1918] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:28:42.404922 19514 log.cc:1196] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/95db349be65b451fad15353507dee9c9
I20260619 02:28:42.405244 19514 ts_tablet_manager.cc:1939] T 95db349be65b451fad15353507dee9c9 P af33c0fb4bcd4d7a91b666d5156c98e6: Deleting consensus metadata
I20260619 02:28:42.406167 19512 ts_tablet_manager.cc:1918] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:28:42.406266 19512 log.cc:1196] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/95db349be65b451fad15353507dee9c9
I20260619 02:28:42.406406 19006 catalog_manager.cc:4841] TS af33c0fb4bcd4d7a91b666d5156c98e6 (127.0.98.3:35473): tablet 95db349be65b451fad15353507dee9c9 (table test-table [id=a868a0fc41e84ac099c9cd495ab21ec7]) successfully deleted
I20260619 02:28:42.406590 19512 ts_tablet_manager.cc:1939] T 95db349be65b451fad15353507dee9c9 P c9332ad5891d48ecbcb04eacb7d9dd59: Deleting consensus metadata
I20260619 02:28:42.407519 19007 catalog_manager.cc:4841] TS c9332ad5891d48ecbcb04eacb7d9dd59 (127.0.98.2:44235): tablet 95db349be65b451fad15353507dee9c9 (table test-table [id=a868a0fc41e84ac099c9cd495ab21ec7]) successfully deleted
W20260619 02:28:42.409772 19008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43227 (error 108)
I20260619 02:28:42.410490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 19213
I20260619 02:28:42.410724 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:42.410405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43227 (local address 127.0.0.1:54620)
0619 02:28:42.410536 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:42.410628 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 02:28:42.412019 19008 catalog_manager.cc:4584] TS a21475a10e414243a716d6dcf0ca0a87 (127.0.98.1:43227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95db349be65b451fad15353507dee9c9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43227: connect: Connection refused (error 111)
I20260619 02:28:42.416175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 19344
I20260619 02:28:42.421437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 18991
2026-06-19T02:28:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1139 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:28:42 dist-test-slave-xkrt krb5kdc[19518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:28:42 dist-test-slave-xkrt krb5kdc[19518](info): set up 2 sockets
Jun 19 02:28:42 dist-test-slave-xkrt krb5kdc[19518](info): commencing operation
krb5kdc: starting...
W20260619 02:28:43.465512 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:28:43 dist-test-slave-xkrt krb5kdc[19518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:28:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:28:43.582630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39201
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:43.689044 19531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:43.689314 19531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:43.689392 19531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:43.693284 19531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:43.693359 19531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:43.693398 19531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:43.693449 19531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:43.693488 19531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:43.698486 19531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:43.699431 19531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:43.700357 19531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:43.708274 19536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:43.708400 19531 server_base.cc:1034] running on GCE node
W20260619 02:28:43.708292 19539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:43.708278 19537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:43.709015 19531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:43.710031 19531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:43.711194 19531 hybrid_clock.cc:648] HybridClock initialized: now 1781836123711174 us; error 48 us; skew 500 ppm
Jun 19 02:28:43 dist-test-slave-xkrt krb5kdc[19518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:43.713805 19531 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:28:43.714640 19531 webserver.cc:460] Webserver started at http://127.0.98.62:39137/ using document root <none> and password file <none>
I20260619 02:28:43.715140 19531 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:43.715205 19531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:43.715416 19531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:43.717317 19531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "eaf54edab8aa495b85ba7c390fd07d2f"
format_stamp: "Formatted at 2026-06-19 02:28:43 on dist-test-slave-xkrt"
server_key: "b1446fd82ab89ebf78072139a8c2dcc2"
server_key_iv: "f83e140bad8e68d90b74aea559098d4b"
server_key_version: "encryptionkey@0"
I20260619 02:28:43.717904 19531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "eaf54edab8aa495b85ba7c390fd07d2f"
format_stamp: "Formatted at 2026-06-19 02:28:43 on dist-test-slave-xkrt"
server_key: "b1446fd82ab89ebf78072139a8c2dcc2"
server_key_iv: "f83e140bad8e68d90b74aea559098d4b"
server_key_version: "encryptionkey@0"
I20260619 02:28:43.721292 19531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:28:43.723544 19546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:43.724694 19531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:28:43.724823 19531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "eaf54edab8aa495b85ba7c390fd07d2f"
format_stamp: "Formatted at 2026-06-19 02:28:43 on dist-test-slave-xkrt"
server_key: "b1446fd82ab89ebf78072139a8c2dcc2"
server_key_iv: "f83e140bad8e68d90b74aea559098d4b"
server_key_version: "encryptionkey@0"
I20260619 02:28:43.724923 19531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:43.735085 19531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:43.738009 19531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:43.738199 19531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:43.744820 19531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39201
I20260619 02:28:43.744846 19598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39201 every 8 connection(s)
I20260619 02:28:43.745759 19531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:43.747473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 19531
I20260619 02:28:43.747601 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:43.747853 392 external_mini_cluster.cc:1428] Setting key 9b6e45f20092b495522d0b1382e8f6e8
I20260619 02:28:43.748448 19599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:43.753935 19599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f: Bootstrap starting.
Jun 19 02:28:43 dist-test-slave-xkrt krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:28:43.756263 19599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:43.756951 19599 log.cc:824] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:43.759060 19599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f: No bootstrap required, opened a new log
I20260619 02:28:43.761665 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:43.749135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.0.98.62:39201)
0619 02:28:43.749578 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:43.749596 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:43.749638 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:28:43.750257 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:43.750273 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:43.750561 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:43.750874 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:43.751634 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:43.752586 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:43.753433 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:43.753983 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:43.754151 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:43.756752 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:43.756757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:43.756769 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:43.756793 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:43.758736 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:43.759145 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:43.759151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:43.759156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:43.759282 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:43.759536 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:43.759539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:43.759541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:43.760002 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:43.760172 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:43.760389 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 02:28:43.760671 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":136,"threads_started":1}
I20260619 02:28:43.763264 19599 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf54edab8aa495b85ba7c390fd07d2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39201 } }
I20260619 02:28:43.763509 19599 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:43.763574 19599 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf54edab8aa495b85ba7c390fd07d2f, State: Initialized, Role: FOLLOWER
I20260619 02:28:43.764009 19599 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf54edab8aa495b85ba7c390fd07d2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39201 } }
I20260619 02:28:43.764134 19599 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:43.764187 19599 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:43.764250 19599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:43.765008 19599 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf54edab8aa495b85ba7c390fd07d2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39201 } }
I20260619 02:28:43.765286 19599 leader_election.cc:302] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaf54edab8aa495b85ba7c390fd07d2f; no voters:
I20260619 02:28:43.765511 19599 leader_election.cc:288] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:43.765650 19604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:43.765851 19604 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [term 1 LEADER]: Becoming Leader. State: Replica: eaf54edab8aa495b85ba7c390fd07d2f, State: Running, Role: LEADER
I20260619 02:28:43.766155 19604 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf54edab8aa495b85ba7c390fd07d2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39201 } }
I20260619 02:28:43.766660 19599 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:43.767583 19605 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaf54edab8aa495b85ba7c390fd07d2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf54edab8aa495b85ba7c390fd07d2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39201 } } }
I20260619 02:28:43.767712 19605 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:43.768024 19606 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaf54edab8aa495b85ba7c390fd07d2f. Latest consensus state: current_term: 1 leader_uuid: "eaf54edab8aa495b85ba7c390fd07d2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf54edab8aa495b85ba7c390fd07d2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39201 } } }
I20260619 02:28:43.768131 19606 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:43.768373 19613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:43.771348 19613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:43.776706 19613 catalog_manager.cc:1347] Generated new cluster ID: 94ed605a0fcd42c992e1e09e633fc473
I20260619 02:28:43.776772 19613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:43.785581 19613 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:43.786724 19613 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:43.796806 19613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f: Generated new TSK 0
I20260619 02:28:43.797611 19613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 02:28:43.828217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39201
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:43.934342 19625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:43.934594 19625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:43.934665 19625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:43.938606 19625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:43.938679 19625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:43.938766 19625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:28:43.943717 19625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:43.944711 19625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:43.945758 19625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:43.954845 19631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:43.954909 19625 server_base.cc:1034] running on GCE node
W20260619 02:28:43.954845 19630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:43.955032 19633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:43.955371 19625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:43.955961 19625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:43.957134 19625 hybrid_clock.cc:648] HybridClock initialized: now 1781836123957104 us; error 37 us; skew 500 ppm
Jun 19 02:28:43 dist-test-slave-xkrt krb5kdc[19518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:43.959812 19625 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 02:28:43.960594 19625 webserver.cc:460] Webserver started at http://127.0.98.1:35377/ using document root <none> and password file <none>
I20260619 02:28:43.961133 19625 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:43.961213 19625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:43.961426 19625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:43.963446 19625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "2612cca4cf8f4ba9a1c078270ba587cf"
format_stamp: "Formatted at 2026-06-19 02:28:43 on dist-test-slave-xkrt"
server_key: "a1e47b4ba8c88407418cb24fd0840f97"
server_key_iv: "0a076c7ac0dd017c446690e27b10568f"
server_key_version: "encryptionkey@0"
I20260619 02:28:43.963992 19625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2612cca4cf8f4ba9a1c078270ba587cf"
format_stamp: "Formatted at 2026-06-19 02:28:43 on dist-test-slave-xkrt"
server_key: "a1e47b4ba8c88407418cb24fd0840f97"
server_key_iv: "0a076c7ac0dd017c446690e27b10568f"
server_key_version: "encryptionkey@0"
I20260619 02:28:43.967569 19625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:28:43.969820 19640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:43.970930 19625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:28:43.971055 19625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "2612cca4cf8f4ba9a1c078270ba587cf"
format_stamp: "Formatted at 2026-06-19 02:28:43 on dist-test-slave-xkrt"
server_key: "a1e47b4ba8c88407418cb24fd0840f97"
server_key_iv: "0a076c7ac0dd017c446690e27b10568f"
server_key_version: "encryptionkey@0"
I20260619 02:28:43.971159 19625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:43.988250 19625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:43.991365 19625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:43.991557 19625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:43.992094 19625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:43.992856 19625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:43.992920 19625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:43.992987 19625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:43.993026 19625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:44.001469 19625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37153
I20260619 02:28:44.001500 19753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37153 every 8 connection(s)
I20260619 02:28:44.002337 19625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:28:44.003896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 19625
I20260619 02:28:44.004047 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:28:44.004395 392 external_mini_cluster.cc:1428] Setting key 8bce516182e2ae2d6ba69865faae25bd
Jun 19 02:28:44 dist-test-slave-xkrt krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 02:28:44.016003 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.004261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58277 (local address 127.0.98.62:39201)
0619 02:28:44.004410 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:44.004415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:44.005296 (+ 881us) server_negotiation.cc:407] Connection header received
0619 02:28:44.006356 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:44.006362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:44.006420 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:44.006511 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:44.007605 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.008148 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.009253 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.009498 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.009655 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.012313 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:44.012317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:44.012320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:44.012346 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:44.013936 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:44.014440 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:44.014443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:44.014444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:44.014522 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:44.014799 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:44.014801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:44.014802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:44.014958 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:44.015040 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:44.015735 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 02:28:44.015867 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:28:44.016876 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.004631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39201 (local address 127.0.98.1:58277)
0619 02:28:44.005112 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:44.005188 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:44.006158 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:44.006656 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:44.006662 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:44.006976 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:44.007458 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.007476 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.008272 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.008275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.009131 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.009139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.009681 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.009685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.009855 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.010393 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:28:44.010409 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:44.012143 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:44.014041 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:44.014045 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:44.014059 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:44.014347 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:44.014602 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:44.014604 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:44.014605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:44.014724 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:44.015151 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:44.015156 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:44.015500 (+ 344us) client_negotiation.cc:769] Sending connection context
0619 02:28:44.015686 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 02:28:44.016042 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":126,"threads_started":1}
I20260619 02:28:44.018193 19754 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39201
I20260619 02:28:44.018450 19754 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:44.018893 19754 heartbeater.cc:511] Master 127.0.98.62:39201 requested a full tablet report, sending...
I20260619 02:28:44.020313 19563 ts_manager.cc:194] Registered new tserver with Master: 2612cca4cf8f4ba9a1c078270ba587cf (127.0.98.1:37153)
I20260619 02:28:44.021432 19563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 02:28:44.032663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39201
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:44.138270 19759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:44.138522 19759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:44.138607 19759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:44.142540 19759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:44.142609 19759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:44.142691 19759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:28:44.147655 19759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:44.148622 19759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:44.149531 19759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:44.158485 19764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:44.158571 19759 server_base.cc:1034] running on GCE node
W20260619 02:28:44.158563 19767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:44.158485 19765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:44.159113 19759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:44.159682 19759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:44.160848 19759 hybrid_clock.cc:648] HybridClock initialized: now 1781836124160830 us; error 30 us; skew 500 ppm
Jun 19 02:28:44 dist-test-slave-xkrt krb5kdc[19518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:44.163475 19759 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 02:28:44.164314 19759 webserver.cc:460] Webserver started at http://127.0.98.2:36553/ using document root <none> and password file <none>
I20260619 02:28:44.164803 19759 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:44.164865 19759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:44.165019 19759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:44.166993 19759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "7134d98c61ce4a37bec2df0665444a41"
format_stamp: "Formatted at 2026-06-19 02:28:44 on dist-test-slave-xkrt"
server_key: "1b5ff32e2cc7ba1ebf0a37d96477eda7"
server_key_iv: "aa3e2485488a33d949874c16f0105fe4"
server_key_version: "encryptionkey@0"
I20260619 02:28:44.167583 19759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7134d98c61ce4a37bec2df0665444a41"
format_stamp: "Formatted at 2026-06-19 02:28:44 on dist-test-slave-xkrt"
server_key: "1b5ff32e2cc7ba1ebf0a37d96477eda7"
server_key_iv: "aa3e2485488a33d949874c16f0105fe4"
server_key_version: "encryptionkey@0"
I20260619 02:28:44.171151 19759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:28:44.173354 19774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:44.174551 19759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:28:44.174731 19759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "7134d98c61ce4a37bec2df0665444a41"
format_stamp: "Formatted at 2026-06-19 02:28:44 on dist-test-slave-xkrt"
server_key: "1b5ff32e2cc7ba1ebf0a37d96477eda7"
server_key_iv: "aa3e2485488a33d949874c16f0105fe4"
server_key_version: "encryptionkey@0"
I20260619 02:28:44.174825 19759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:44.189168 19759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:44.192461 19759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:44.192607 19759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:44.193094 19759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:44.193894 19759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:44.193938 19759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:44.193976 19759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:44.193992 19759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:44.203562 19759 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42669
I20260619 02:28:44.203604 19887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42669 every 8 connection(s)
I20260619 02:28:44.204406 19759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:28:44.208073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 19759
I20260619 02:28:44.208220 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:28:44.208534 392 external_mini_cluster.cc:1428] Setting key 3175d90406ed903495201df34e5dc78d
Jun 19 02:28:44 dist-test-slave-xkrt krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:28:44.218655 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.206126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41781 (local address 127.0.98.62:39201)
0619 02:28:44.206260 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:44.206264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:44.207082 (+ 818us) server_negotiation.cc:407] Connection header received
0619 02:28:44.207941 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:44.207943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:44.207988 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:44.208061 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:44.209517 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.210225 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.211999 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.212238 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.212343 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.214925 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:44.214931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:44.214937 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:44.214973 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:44.216584 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:44.217065 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:44.217069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:44.217071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:44.217183 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:44.217495 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:44.217501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:44.217504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:44.217759 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:44.217871 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:44.218343 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 02:28:44.218467 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:28:44.219332 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.206496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39201 (local address 127.0.98.2:41781)
0619 02:28:44.206920 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:44.206959 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:44.207717 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:44.208268 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:44.208278 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:44.208765 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:44.209375 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.209390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.210342 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.210345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.211868 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.211881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.212376 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.212380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.212550 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.213069 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:28:44.213094 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:44.214764 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:44.216701 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:44.216705 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:44.216716 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:44.216956 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:44.217279 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:44.217281 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:44.217282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:44.217404 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:44.217874 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:44.217879 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:44.218105 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 02:28:44.218266 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 02:28:44.218499 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":110,"threads_started":1}
I20260619 02:28:44.220552 19888 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39201
I20260619 02:28:44.220820 19888 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 02:28:44.221357 19888 heartbeater.cc:511] Master 127.0.98.62:39201 requested a full tablet report, sending...
I20260619 02:28:44.222494 19563 ts_manager.cc:194] Registered new tserver with Master: 7134d98c61ce4a37bec2df0665444a41 (127.0.98.2:42669)
I20260619 02:28:44.223138 19563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 02:28:44.236946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39201
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:44.344969 19893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:44.345208 19893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:44.345249 19893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:44.349304 19893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:44.349355 19893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:44.349452 19893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:28:44.354532 19893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:44.355504 19893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:44.356448 19893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:44.365430 19898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:44.365453 19901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:44.365448 19899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:44.365648 19893 server_base.cc:1034] running on GCE node
I20260619 02:28:44.366051 19893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:44.366726 19893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:44.367893 19893 hybrid_clock.cc:648] HybridClock initialized: now 1781836124367879 us; error 26 us; skew 500 ppm
Jun 19 02:28:44 dist-test-slave-xkrt krb5kdc[19518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:44.370586 19893 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 02:28:44.371315 19893 webserver.cc:460] Webserver started at http://127.0.98.3:37059/ using document root <none> and password file <none>
I20260619 02:28:44.371788 19893 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:44.371845 19893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:44.372042 19893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:44.373934 19893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "d6270ff293cf4aac9ccb47fe4820c17a"
format_stamp: "Formatted at 2026-06-19 02:28:44 on dist-test-slave-xkrt"
server_key: "0b263edbd136afd8a5d9e805c57d35b1"
server_key_iv: "a0682b43d090518aed17796ed1a60d03"
server_key_version: "encryptionkey@0"
I20260619 02:28:44.374490 19893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d6270ff293cf4aac9ccb47fe4820c17a"
format_stamp: "Formatted at 2026-06-19 02:28:44 on dist-test-slave-xkrt"
server_key: "0b263edbd136afd8a5d9e805c57d35b1"
server_key_iv: "a0682b43d090518aed17796ed1a60d03"
server_key_version: "encryptionkey@0"
I20260619 02:28:44.377959 19893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:28:44.380398 19908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:44.381623 19893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:28:44.381742 19893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "d6270ff293cf4aac9ccb47fe4820c17a"
format_stamp: "Formatted at 2026-06-19 02:28:44 on dist-test-slave-xkrt"
server_key: "0b263edbd136afd8a5d9e805c57d35b1"
server_key_iv: "a0682b43d090518aed17796ed1a60d03"
server_key_version: "encryptionkey@0"
I20260619 02:28:44.381835 19893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:44.408756 19893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:44.411974 19893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:44.412161 19893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:44.412757 19893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:44.413589 19893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:44.413632 19893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:44.413690 19893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:44.413723 19893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:44.422098 19893 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45717
I20260619 02:28:44.422142 20021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45717 every 8 connection(s)
I20260619 02:28:44.422933 19893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:28:44 dist-test-slave-xkrt krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:28:44.432983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 19893
I20260619 02:28:44.433099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:28:44.433420 392 external_mini_cluster.cc:1428] Setting key 210c14f1fb1c85f28ff3c22fef571f9b
I20260619 02:28:44.435781 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.424669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57957 (local address 127.0.98.62:39201)
0619 02:28:44.424797 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:44.424800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:44.425595 (+ 795us) server_negotiation.cc:407] Connection header received
0619 02:28:44.426474 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:44.426477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:44.426533 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:44.426629 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:44.427790 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.428375 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.429515 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.429794 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.429902 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.432187 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:44.432190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:44.432192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:44.432216 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:44.433722 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:44.434242 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:44.434245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:44.434246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:44.434355 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:44.434751 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:44.434755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:44.434761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:44.434976 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:44.435073 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:44.435558 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 02:28:44.435668 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:28:44.436609 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.424997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39201 (local address 127.0.98.3:57957)
0619 02:28:44.425420 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:44.425459 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:44.426278 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:44.426817 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:44.426825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:44.427159 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:44.427649 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.427662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.428494 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.428501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.429392 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.429401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.429935 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.429939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.430111 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.430633 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:28:44.430649 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:44.432017 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:44.433842 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:44.433848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:44.433861 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:44.434153 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:44.434452 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:28:44.434456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:28:44.434457 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:28:44.434630 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:28:44.435075 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:44.435080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:44.435296 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 02:28:44.435490 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 02:28:44.435781 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":120,"threads_started":1}
I20260619 02:28:44.437713 20022 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39201
I20260619 02:28:44.437937 20022 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:44.438396 20022 heartbeater.cc:511] Master 127.0.98.62:39201 requested a full tablet report, sending...
I20260619 02:28:44.439388 19563 ts_manager.cc:194] Registered new tserver with Master: d6270ff293cf4aac9ccb47fe4820c17a (127.0.98.3:45717)
I20260619 02:28:44.439958 19563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57957
I20260619 02:28:44.447170 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:28:44.455737 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.449104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.0.98.62:39201)
0619 02:28:44.449253 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:44.449257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:44.449431 (+ 174us) server_negotiation.cc:407] Connection header received
0619 02:28:44.449520 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:44.449522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:44.449584 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:44.449648 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:44.450271 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.450771 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.451561 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.451784 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.451879 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.453108 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:44.453111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:44.453112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:44.453131 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:44.454314 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:44.454655 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:44.454657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:44.454658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:44.454725 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:44.454928 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:44.454930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:44.454931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:44.455144 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:44.455242 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:44.455434 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 02:28:44.455524 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:28:44.458343 19563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:28:44.461244 19563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:28:44.475502 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.469886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45210 (local address 127.0.98.2:42669)
0619 02:28:44.470405 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:44.470416 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:44.470438 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:28:44.470514 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:44.470519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:44.470728 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:44.470884 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:44.471454 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.472636 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.474150 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.474913 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.475024 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.475173 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:44.475255 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 02:28:44.475346 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":102,"threads_started":1}
I20260619 02:28:44.475540 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.469296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42669 (local address 127.0.0.1:45210)
0619 02:28:44.469919 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:44.469938 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:44.470062 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:44.471032 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:44.471034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:44.471045 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:44.471315 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.471322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.472789 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.472792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.473998 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.474008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.475022 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.475024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.475101 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.475128 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 02:28:44.475223 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 02:28:44.475341 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":115,"spinlock_wait_cycles":1408,"thread_start_us":94,"threads_started":1}
I20260619 02:28:44.475721 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.469985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45717 (local address 127.0.0.1:50848)
0619 02:28:44.470920 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:44.470934 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:44.471012 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:44.471423 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:44.471426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:44.471441 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:44.471539 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.471546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.472482 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.472487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.473972 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.473978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.475312 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.475315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.475390 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.475413 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:28:44.475524 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:28:44.475603 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":95,"thread_start_us":80,"threads_started":1}
I20260619 02:28:44.476861 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.469209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37153 (local address 127.0.0.1:41854)
0619 02:28:44.469879 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:44.470418 (+ 539us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:44.470530 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:44.471005 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:44.471013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:44.471038 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:44.471317 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.471322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.472766 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.472769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.473963 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.473974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.476460 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.476464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.476556 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.476587 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:28:44.476655 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 02:28:44.476731 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":75,"spinlock_wait_cycles":8064,"thread_start_us":80,"threads_started":1}
I20260619 02:28:44.476930 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.470134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.0.98.3:45717)
0619 02:28:44.470764 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:44.470775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:44.471099 (+ 324us) server_negotiation.cc:407] Connection header received
0619 02:28:44.471171 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:44.471176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:44.471305 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:44.471441 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:44.471661 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.472365 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.474403 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.475199 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.475304 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.476626 (+ 1322us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:44.476693 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 02:28:44.476791 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":142,"threads_started":1}
I20260619 02:28:44.477288 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.469373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.0.98.1:37153)
0619 02:28:44.470318 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:44.470332 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:44.470505 (+ 173us) server_negotiation.cc:407] Connection header received
0619 02:28:44.470641 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:44.470647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:44.470844 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:44.470998 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:44.471453 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.472644 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.474081 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.476351 (+ 2270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.476464 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.476584 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:44.477032 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 02:28:44.477131 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":107,"threads_started":1}
I20260619 02:28:44.478878 19822 tablet_service.cc:1467] Processing CreateTablet for tablet d89825251dbe41c79a60b85a73b394eb (DEFAULT_TABLE table=test-table [id=e125479ff1f64deba7f3000ac3c91a7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:44.479782 19822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d89825251dbe41c79a60b85a73b394eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:44.479646 19956 tablet_service.cc:1467] Processing CreateTablet for tablet d89825251dbe41c79a60b85a73b394eb (DEFAULT_TABLE table=test-table [id=e125479ff1f64deba7f3000ac3c91a7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:44.479907 19688 tablet_service.cc:1467] Processing CreateTablet for tablet d89825251dbe41c79a60b85a73b394eb (DEFAULT_TABLE table=test-table [id=e125479ff1f64deba7f3000ac3c91a7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:44.480651 19956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d89825251dbe41c79a60b85a73b394eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:44.480746 19688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d89825251dbe41c79a60b85a73b394eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:44.486318 20039 tablet_bootstrap.cc:492] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41: Bootstrap starting.
I20260619 02:28:44.487763 20040 tablet_bootstrap.cc:492] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf: Bootstrap starting.
I20260619 02:28:44.488145 20041 tablet_bootstrap.cc:492] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a: Bootstrap starting.
I20260619 02:28:44.488557 20039 tablet_bootstrap.cc:654] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:44.489918 20039 log.cc:824] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:44.490401 20041 tablet_bootstrap.cc:654] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:44.491457 20040 tablet_bootstrap.cc:654] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:44.491581 20041 log.cc:824] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:44.492559 20040 log.cc:824] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:44.492682 20039 tablet_bootstrap.cc:492] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41: No bootstrap required, opened a new log
I20260619 02:28:44.492877 20039 ts_tablet_manager.cc:1397] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 02:28:44.493278 20041 tablet_bootstrap.cc:492] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a: No bootstrap required, opened a new log
I20260619 02:28:44.493449 20041 ts_tablet_manager.cc:1397] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:28:44.494427 20040 tablet_bootstrap.cc:492] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf: No bootstrap required, opened a new log
I20260619 02:28:44.494597 20040 ts_tablet_manager.cc:1397] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 02:28:44.496340 20041 raft_consensus.cc:348] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "7134d98c61ce4a37bec2df0665444a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42669 } } peers { permanent_uuid: "d6270ff293cf4aac9ccb47fe4820c17a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45717 } }
I20260619 02:28:44.496582 20041 raft_consensus.cc:374] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:44.496640 20041 raft_consensus.cc:729] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6270ff293cf4aac9ccb47fe4820c17a, State: Initialized, Role: FOLLOWER
I20260619 02:28:44.496701 20039 raft_consensus.cc:348] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "7134d98c61ce4a37bec2df0665444a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42669 } } peers { permanent_uuid: "d6270ff293cf4aac9ccb47fe4820c17a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45717 } }
I20260619 02:28:44.496953 20039 raft_consensus.cc:374] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:44.497006 20039 raft_consensus.cc:729] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7134d98c61ce4a37bec2df0665444a41, State: Initialized, Role: FOLLOWER
I20260619 02:28:44.497087 20041 consensus_queue.cc:260] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "7134d98c61ce4a37bec2df0665444a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42669 } } peers { permanent_uuid: "d6270ff293cf4aac9ccb47fe4820c17a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45717 } }
I20260619 02:28:44.497398 20039 consensus_queue.cc:260] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "7134d98c61ce4a37bec2df0665444a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42669 } } peers { permanent_uuid: "d6270ff293cf4aac9ccb47fe4820c17a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45717 } }
I20260619 02:28:44.498109 20041 ts_tablet_manager.cc:1428] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:28:44.498306 20039 ts_tablet_manager.cc:1428] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 02:28:44.498322 20022 heartbeater.cc:503] Master 127.0.98.62:39201 was elected leader, sending a full tablet report...
I20260619 02:28:44.498297 20040 raft_consensus.cc:348] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "7134d98c61ce4a37bec2df0665444a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42669 } } peers { permanent_uuid: "d6270ff293cf4aac9ccb47fe4820c17a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45717 } }
I20260619 02:28:44.498526 20040 raft_consensus.cc:374] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:44.498582 20040 raft_consensus.cc:729] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2612cca4cf8f4ba9a1c078270ba587cf, State: Initialized, Role: FOLLOWER
I20260619 02:28:44.498641 19888 heartbeater.cc:503] Master 127.0.98.62:39201 was elected leader, sending a full tablet report...
I20260619 02:28:44.498926 20040 consensus_queue.cc:260] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "7134d98c61ce4a37bec2df0665444a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42669 } } peers { permanent_uuid: "d6270ff293cf4aac9ccb47fe4820c17a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45717 } }
I20260619 02:28:44.499457 19754 heartbeater.cc:503] Master 127.0.98.62:39201 was elected leader, sending a full tablet report...
I20260619 02:28:44.500643 20040 ts_tablet_manager.cc:1428] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 02:28:44.503463 19755 tablet.cc:2369] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:28:44.673853 20023 tablet.cc:2369] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:28:44.705472 19889 tablet.cc:2369] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:28:44.772822 20047 raft_consensus.cc:482] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:28:44.773013 20047 raft_consensus.cc:504] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "7134d98c61ce4a37bec2df0665444a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42669 } } peers { permanent_uuid: "d6270ff293cf4aac9ccb47fe4820c17a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45717 } }
I20260619 02:28:44.774255 20047 leader_election.cc:288] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7134d98c61ce4a37bec2df0665444a41 (127.0.98.2:42669), d6270ff293cf4aac9ccb47fe4820c17a (127.0.98.3:45717)
I20260619 02:28:44.778218 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.774605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46073 (local address 127.0.98.3:45717)
0619 02:28:44.774767 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:44.774772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:44.775052 (+ 280us) server_negotiation.cc:407] Connection header received
0619 02:28:44.775176 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:44.775180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:44.775235 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:44.775326 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:44.775577 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.776051 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.777169 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.777628 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.777768 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.777840 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:44.778013 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 02:28:44.778103 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 02:28:44.778223 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.774488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45717 (local address 127.0.98.1:46073)
0619 02:28:44.774928 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:44.774939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:44.775063 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:44.775339 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:44.775341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:44.775355 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:44.775468 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.775474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.776152 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.776158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.777049 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.777058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.777771 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.777775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.777890 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.777912 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:28:44.777985 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 02:28:44.778047 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":52,"thread_start_us":135,"threads_started":1}
I20260619 02:28:44.778314 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.774584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36743 (local address 127.0.98.2:42669)
0619 02:28:44.774819 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:44.774824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:44.774864 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:28:44.775030 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:44.775034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:44.775109 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:44.775191 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:44.775485 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.776074 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.777199 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.777769 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.777873 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.777930 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:44.778115 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 02:28:44.778184 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260619 02:28:44.778473 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.774488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42669 (local address 127.0.98.1:36743)
0619 02:28:44.774785 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:44.774800 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:44.774913 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:44.775206 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:44.775209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:44.775225 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:44.775373 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.775380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.776201 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.776203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.777048 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:44.777057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.777879 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.777883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:44.777988 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.778011 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:28:44.778087 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:28:44.778150 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":19,"thread_start_us":73,"threads_started":1}
I20260619 02:28:44.779003 19842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d89825251dbe41c79a60b85a73b394eb" candidate_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7134d98c61ce4a37bec2df0665444a41" is_pre_election: true
I20260619 02:28:44.779018 19976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d89825251dbe41c79a60b85a73b394eb" candidate_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6270ff293cf4aac9ccb47fe4820c17a" is_pre_election: true
I20260619 02:28:44.779282 19842 raft_consensus.cc:2454] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2612cca4cf8f4ba9a1c078270ba587cf in term 0.
I20260619 02:28:44.779294 19976 raft_consensus.cc:2454] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2612cca4cf8f4ba9a1c078270ba587cf in term 0.
I20260619 02:28:44.779722 19642 leader_election.cc:302] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2612cca4cf8f4ba9a1c078270ba587cf, d6270ff293cf4aac9ccb47fe4820c17a; no voters:
I20260619 02:28:44.779955 20047 raft_consensus.cc:2781] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:28:44.780032 20047 raft_consensus.cc:482] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:28:44.780092 20047 raft_consensus.cc:3037] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:44.781101 20047 raft_consensus.cc:504] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "7134d98c61ce4a37bec2df0665444a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42669 } } peers { permanent_uuid: "d6270ff293cf4aac9ccb47fe4820c17a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45717 } }
I20260619 02:28:44.781399 20047 leader_election.cc:288] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [CANDIDATE]: Term 1 election: Requested vote from peers 7134d98c61ce4a37bec2df0665444a41 (127.0.98.2:42669), d6270ff293cf4aac9ccb47fe4820c17a (127.0.98.3:45717)
I20260619 02:28:44.781759 19842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d89825251dbe41c79a60b85a73b394eb" candidate_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7134d98c61ce4a37bec2df0665444a41"
I20260619 02:28:44.781759 19976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d89825251dbe41c79a60b85a73b394eb" candidate_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6270ff293cf4aac9ccb47fe4820c17a"
I20260619 02:28:44.781878 19976 raft_consensus.cc:3037] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:44.781908 19842 raft_consensus.cc:3037] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:44.783130 19976 raft_consensus.cc:2454] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2612cca4cf8f4ba9a1c078270ba587cf in term 1.
I20260619 02:28:44.783130 19842 raft_consensus.cc:2454] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2612cca4cf8f4ba9a1c078270ba587cf in term 1.
I20260619 02:28:44.783404 19642 leader_election.cc:302] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2612cca4cf8f4ba9a1c078270ba587cf, d6270ff293cf4aac9ccb47fe4820c17a; no voters:
I20260619 02:28:44.783571 20047 raft_consensus.cc:2781] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:44.783756 20047 raft_consensus.cc:686] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [term 1 LEADER]: Becoming Leader. State: Replica: 2612cca4cf8f4ba9a1c078270ba587cf, State: Running, Role: LEADER
I20260619 02:28:44.784083 20047 consensus_queue.cc:237] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } } peers { permanent_uuid: "7134d98c61ce4a37bec2df0665444a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42669 } } peers { permanent_uuid: "d6270ff293cf4aac9ccb47fe4820c17a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45717 } }
I20260619 02:28:44.787315 19562 catalog_manager.cc:5495] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2612cca4cf8f4ba9a1c078270ba587cf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2612cca4cf8f4ba9a1c078270ba587cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7134d98c61ce4a37bec2df0665444a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6270ff293cf4aac9ccb47fe4820c17a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20260619 02:28:44.835306 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.831976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41862 (local address 127.0.98.1:37153)
0619 02:28:44.832109 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:44.832113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:44.832126 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:28:44.832197 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:44.832199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:44.832252 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:44.832340 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:28:44.832536 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.833036 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.833897 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:44.834236 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:44.834364 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:44.834598 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:28:44.835001 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:28:44.835125 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:44.835152 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 02:28:44.835200 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:28:44.841097 19976 raft_consensus.cc:1261] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a [term 1 FOLLOWER]: Refusing update from remote peer 2612cca4cf8f4ba9a1c078270ba587cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:28:44.841102 19842 raft_consensus.cc:1261] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41 [term 1 FOLLOWER]: Refusing update from remote peer 2612cca4cf8f4ba9a1c078270ba587cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:28:44.841830 20047 consensus_queue.cc:1035] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7134d98c61ce4a37bec2df0665444a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:28:44.842056 20050 consensus_queue.cc:1035] T d89825251dbe41c79a60b85a73b394eb P 2612cca4cf8f4ba9a1c078270ba587cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6270ff293cf4aac9ccb47fe4820c17a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:28:44.852226 20055 mvcc.cc:205] Tried to move back new op lower bound from 7298400767341334528 to 7298400767118217216. Current Snapshot: MvccSnapshot[applied={T|T < 7298400767341334528 or (T in {7298400767341334528})}]
I20260619 02:28:44.856827 19561 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:28:44.856962 19561 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:28:44.859176 19561 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eaf54edab8aa495b85ba7c390fd07d2f: Sending DeleteTablet for 3 replicas of tablet d89825251dbe41c79a60b85a73b394eb
I20260619 02:28:44.859979 19822 tablet_service.cc:1514] Processing DeleteTablet for tablet d89825251dbe41c79a60b85a73b394eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45210
I20260619 02:28:44.860080 19688 tablet_service.cc:1514] Processing DeleteTablet for tablet d89825251dbe41c79a60b85a73b394eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41854
I20260619 02:28:44.860450 20062 tablet_replica.cc:330] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41: stopping tablet replica
I20260619 02:28:44.860523 19956 tablet_service.cc:1514] Processing DeleteTablet for tablet d89825251dbe41c79a60b85a73b394eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50848
I20260619 02:28:44.860864 20062 raft_consensus.cc:2229] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:28:44.860893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 19625
I20260619 02:28:44.861213 20062 raft_consensus.cc:2258] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:28:44.862835 20062 ts_tablet_manager.cc:1905] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:28:44.864953 20062 ts_tablet_manager.cc:1918] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:28:44.865029 20062 log.cc:1196] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/d89825251dbe41c79a60b85a73b394eb
I20260619 02:28:44.865237 20064 tablet_replica.cc:330] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a: stopping tablet replica
I20260619 02:28:44.865319 20062 ts_tablet_manager.cc:1939] T d89825251dbe41c79a60b85a73b394eb P 7134d98c61ce4a37bec2df0665444a41: Deleting consensus metadata
I20260619 02:28:44.865532 20064 raft_consensus.cc:2229] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:28:44.865842 20064 raft_consensus.cc:2258] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:28:44.866744 19550 catalog_manager.cc:4841] TS 7134d98c61ce4a37bec2df0665444a41 (127.0.98.2:42669): tablet d89825251dbe41c79a60b85a73b394eb (table test-table [id=e125479ff1f64deba7f3000ac3c91a7c]) successfully deleted
I20260619 02:28:44.867190 20064 ts_tablet_manager.cc:1905] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:28:44.868638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 19759
W20260619 02:28:44.868606 19548 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37153 (error 108)
I20260619 02:28:44.870949 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:44.870697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37153 (local address 127.0.0.1:41868)
0619 02:28:44.870815 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:44.870881 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 02:28:44.871131 19548 catalog_manager.cc:4584] TS 2612cca4cf8f4ba9a1c078270ba587cf (127.0.98.1:37153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d89825251dbe41c79a60b85a73b394eb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37153: connect: Connection refused (error 111)
I20260619 02:28:44.871281 20064 ts_tablet_manager.cc:1918] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:28:44.871382 20064 log.cc:1196] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/d89825251dbe41c79a60b85a73b394eb
I20260619 02:28:44.871709 20064 ts_tablet_manager.cc:1939] T d89825251dbe41c79a60b85a73b394eb P d6270ff293cf4aac9ccb47fe4820c17a: Deleting consensus metadata
I20260619 02:28:44.872848 19548 catalog_manager.cc:4841] TS d6270ff293cf4aac9ccb47fe4820c17a (127.0.98.3:45717): tablet d89825251dbe41c79a60b85a73b394eb (table test-table [id=e125479ff1f64deba7f3000ac3c91a7c]) successfully deleted
I20260619 02:28:44.874987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 19893
I20260619 02:28:44.880275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 19531
2026-06-19T02:28:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2459 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T02:28:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:44Z Disabled control of system clock
I20260619 02:28:44.909350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42501
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:28:45.016855 20069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:45.017102 20069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:45.017136 20069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:45.021071 20069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:45.021124 20069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:45.021139 20069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:45.021157 20069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:45.021170 20069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:45.025897 20069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:45.026876 20069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:45.028258 20069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:45.036409 20074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:45.036448 20075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:45.036413 20077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:45.036446 20069 server_base.cc:1034] running on GCE node
I20260619 02:28:45.037060 20069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:45.038136 20069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:45.039291 20069 hybrid_clock.cc:648] HybridClock initialized: now 1781836125039268 us; error 35 us; skew 500 ppm
I20260619 02:28:45.041045 20069 webserver.cc:460] Webserver started at http://127.0.98.62:36723/ using document root <none> and password file <none>
I20260619 02:28:45.041640 20069 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:45.041709 20069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:45.041922 20069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:45.043855 20069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "4c58d563605742169a66da49ff7454f4"
format_stamp: "Formatted at 2026-06-19 02:28:45 on dist-test-slave-xkrt"
server_key: "0735894521c6890a10e43ffb45ce1901"
server_key_iv: "25f323f1f1067ea2d234bd6f6c89011d"
server_key_version: "encryptionkey@0"
I20260619 02:28:45.044402 20069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c58d563605742169a66da49ff7454f4"
format_stamp: "Formatted at 2026-06-19 02:28:45 on dist-test-slave-xkrt"
server_key: "0735894521c6890a10e43ffb45ce1901"
server_key_iv: "25f323f1f1067ea2d234bd6f6c89011d"
server_key_version: "encryptionkey@0"
I20260619 02:28:45.047833 20069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:28:45.050112 20083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:45.051262 20069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:28:45.051389 20069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "4c58d563605742169a66da49ff7454f4"
format_stamp: "Formatted at 2026-06-19 02:28:45 on dist-test-slave-xkrt"
server_key: "0735894521c6890a10e43ffb45ce1901"
server_key_iv: "25f323f1f1067ea2d234bd6f6c89011d"
server_key_version: "encryptionkey@0"
I20260619 02:28:45.051492 20069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:45.076897 20069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:45.077642 20069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:45.077842 20069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:45.084920 20069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42501
I20260619 02:28:45.084934 20135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42501 every 8 connection(s)
I20260619 02:28:45.085886 20069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:45.088970 20136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:45.094375 20136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4: Bootstrap starting.
I20260619 02:28:45.094696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 20069
I20260619 02:28:45.094822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:45.095132 392 external_mini_cluster.cc:1428] Setting key 2d1fa36f0beca3203ace15d16fe4332b
I20260619 02:28:45.097185 20136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:45.097967 20136 log.cc:824] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:45.099856 20136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4: No bootstrap required, opened a new log
I20260619 02:28:45.103396 20136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c58d563605742169a66da49ff7454f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42501 } }
I20260619 02:28:45.103614 20136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:45.103675 20136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c58d563605742169a66da49ff7454f4, State: Initialized, Role: FOLLOWER
I20260619 02:28:45.104089 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:45.096468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.0.98.62:42501)
0619 02:28:45.097101 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:45.097114 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:45.097148 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:28:45.097826 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:45.097843 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:45.098267 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:45.098635 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:45.098989 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.099936 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.100810 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.101299 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.101457 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:45.102296 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:45.102301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:45.102314 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:45.102690 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:45.102747 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:45.102765 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:45.102868 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:45.102979 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 02:28:45.103231 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":134,"threads_started":1}
I20260619 02:28:45.104090 20136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c58d563605742169a66da49ff7454f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42501 } }
I20260619 02:28:45.104277 20136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:45.104341 20136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:45.104419 20136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:45.105314 20136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c58d563605742169a66da49ff7454f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42501 } }
I20260619 02:28:45.105718 20136 leader_election.cc:302] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c58d563605742169a66da49ff7454f4; no voters:
I20260619 02:28:45.106026 20136 leader_election.cc:288] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:45.106122 20141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:45.106354 20141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [term 1 LEADER]: Becoming Leader. State: Replica: 4c58d563605742169a66da49ff7454f4, State: Running, Role: LEADER
I20260619 02:28:45.106662 20141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c58d563605742169a66da49ff7454f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42501 } }
I20260619 02:28:45.107138 20136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:45.108224 20142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c58d563605742169a66da49ff7454f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c58d563605742169a66da49ff7454f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42501 } } }
I20260619 02:28:45.108355 20142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:45.108589 20143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c58d563605742169a66da49ff7454f4. Latest consensus state: current_term: 1 leader_uuid: "4c58d563605742169a66da49ff7454f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c58d563605742169a66da49ff7454f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42501 } } }
I20260619 02:28:45.108723 20143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:45.108698 20150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:45.112023 20150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:45.117422 20150 catalog_manager.cc:1347] Generated new cluster ID: 109be6f32867478181bd858b37960430
I20260619 02:28:45.117496 20150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:45.124847 20150 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:45.125996 20150 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:45.137595 20150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4: Generated new TSK 0
I20260619 02:28:45.138321 20150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:28:45.144886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42501
--builtin_ntp_servers=127.0.98.20:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:28:45.255765 20160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:45.256114 20160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:45.256155 20160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:45.260442 20160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:45.260491 20160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:45.260545 20160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 02:28:45.265089 20160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:45.266109 20160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:45.267390 20160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:45.276491 20165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:45.276501 20168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:45.276494 20166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:45.276575 20160 server_base.cc:1034] running on GCE node
I20260619 02:28:45.277056 20160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:45.277760 20160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:45.278915 20160 hybrid_clock.cc:648] HybridClock initialized: now 1781836125278902 us; error 24 us; skew 500 ppm
I20260619 02:28:45.284701 20160 webserver.cc:460] Webserver started at http://127.0.98.1:40903/ using document root <none> and password file <none>
I20260619 02:28:45.285319 20160 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:45.285384 20160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:45.285686 20160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:45.287688 20160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/data/instance:
uuid: "3c5b53c5b9e54e138db5f87b5dea957c"
format_stamp: "Formatted at 2026-06-19 02:28:45 on dist-test-slave-xkrt"
server_key: "ac1e412f6de07e8e4e0af5c5c8f7e71c"
server_key_iv: "c045c3625d81c42cf81aed77e8975494"
server_key_version: "encryptionkey@0"
I20260619 02:28:45.288224 20160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3c5b53c5b9e54e138db5f87b5dea957c"
format_stamp: "Formatted at 2026-06-19 02:28:45 on dist-test-slave-xkrt"
server_key: "ac1e412f6de07e8e4e0af5c5c8f7e71c"
server_key_iv: "c045c3625d81c42cf81aed77e8975494"
server_key_version: "encryptionkey@0"
I20260619 02:28:45.291853 20160 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:28:45.294186 20174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:45.295258 20160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:28:45.295382 20160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/wal
uuid: "3c5b53c5b9e54e138db5f87b5dea957c"
format_stamp: "Formatted at 2026-06-19 02:28:45 on dist-test-slave-xkrt"
server_key: "ac1e412f6de07e8e4e0af5c5c8f7e71c"
server_key_iv: "c045c3625d81c42cf81aed77e8975494"
server_key_version: "encryptionkey@0"
I20260619 02:28:45.295485 20160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:45.331151 20160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:45.331869 20160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:45.332082 20160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:45.332679 20160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:45.333611 20160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:45.333655 20160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:45.333724 20160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:45.333753 20160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:45.342423 20160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40823
I20260619 02:28:45.342475 20287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40823 every 8 connection(s)
I20260619 02:28:45.343323 20160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/data/info.pb
I20260619 02:28:45.351114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 20160
I20260619 02:28:45.351289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-0/wal/instance
I20260619 02:28:45.351589 392 external_mini_cluster.cc:1428] Setting key 86346b0547ca54a46420dfefe2ddcd36
I20260619 02:28:45.352308 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:45.345109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59997 (local address 127.0.98.62:42501)
0619 02:28:45.345251 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:45.345255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:45.346274 (+ 1019us) server_negotiation.cc:407] Connection header received
0619 02:28:45.347646 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:45.347650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:45.347707 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:45.347782 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:45.348618 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.349170 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.350349 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.350596 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.350708 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:45.351826 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:45.351830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:45.351833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:45.351845 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:45.351871 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:45.351879 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:45.351948 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:45.352103 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 02:28:45.352171 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:28:45.353075 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:45.345464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42501 (local address 127.0.98.1:59997)
0619 02:28:45.346128 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:45.346174 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:45.347430 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:45.347937 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:45.347944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:45.348025 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:45.348484 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:45.348496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.349289 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.349293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:45.350226 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:45.350236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.350747 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.350751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:45.350916 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:45.351541 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:28:45.351543 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:45.351581 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:28:45.351675 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:45.351961 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:45.351972 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:45.351977 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:28:45.352099 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 02:28:45.352360 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":113,"threads_started":1}
I20260619 02:28:45.353575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42501
--builtin_ntp_servers=127.0.98.20:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:28:45.354193 20288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42501
I20260619 02:28:45.354405 20288 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:45.354929 20288 heartbeater.cc:511] Master 127.0.98.62:42501 requested a full tablet report, sending...
I20260619 02:28:45.356387 20100 ts_manager.cc:194] Registered new tserver with Master: 3c5b53c5b9e54e138db5f87b5dea957c (127.0.98.1:40823)
I20260619 02:28:45.357514 20100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59997
W20260619 02:28:45.461616 20291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:45.461853 20291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:45.461887 20291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:45.465837 20291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:45.465878 20291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:45.465973 20291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 02:28:45.470573 20291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:45.471549 20291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:45.472954 20291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:45.482107 20296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:45.482087 20299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:45.482110 20297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:45.482259 20291 server_base.cc:1034] running on GCE node
I20260619 02:28:45.482719 20291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:45.483333 20291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:45.484506 20291 hybrid_clock.cc:648] HybridClock initialized: now 1781836125484493 us; error 25 us; skew 500 ppm
I20260619 02:28:45.486243 20291 webserver.cc:460] Webserver started at http://127.0.98.2:39603/ using document root <none> and password file <none>
I20260619 02:28:45.486791 20291 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:45.486884 20291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:45.487094 20291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:45.489110 20291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/data/instance:
uuid: "bc97fe6a6ee14619b138499cc4f19aab"
format_stamp: "Formatted at 2026-06-19 02:28:45 on dist-test-slave-xkrt"
server_key: "67830048052dd7ab29a150db4227ab70"
server_key_iv: "379c3d392e92eaf19ab94e19b0b0095c"
server_key_version: "encryptionkey@0"
I20260619 02:28:45.489719 20291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bc97fe6a6ee14619b138499cc4f19aab"
format_stamp: "Formatted at 2026-06-19 02:28:45 on dist-test-slave-xkrt"
server_key: "67830048052dd7ab29a150db4227ab70"
server_key_iv: "379c3d392e92eaf19ab94e19b0b0095c"
server_key_version: "encryptionkey@0"
I20260619 02:28:45.493077 20291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:28:45.495397 20305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:45.496556 20291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:28:45.496690 20291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/wal
uuid: "bc97fe6a6ee14619b138499cc4f19aab"
format_stamp: "Formatted at 2026-06-19 02:28:45 on dist-test-slave-xkrt"
server_key: "67830048052dd7ab29a150db4227ab70"
server_key_iv: "379c3d392e92eaf19ab94e19b0b0095c"
server_key_version: "encryptionkey@0"
I20260619 02:28:45.496795 20291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:45.517272 20291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:45.517969 20291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:45.518170 20291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:45.518731 20291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:45.519647 20291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:45.519713 20291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:45.519785 20291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:45.519825 20291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:45.528302 20291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46249
I20260619 02:28:45.528335 20418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46249 every 8 connection(s)
I20260619 02:28:45.529143 20291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/data/info.pb
I20260619 02:28:45.537631 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:45.530925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55507 (local address 127.0.98.62:42501)
0619 02:28:45.531051 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:45.531055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:45.531984 (+ 929us) server_negotiation.cc:407] Connection header received
0619 02:28:45.533316 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:45.533319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:45.533363 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:45.533435 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:45.534285 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.534790 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.535956 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.536172 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.536245 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:45.537218 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:45.537221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:45.537223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:45.537234 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:45.537253 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:45.537260 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:45.537313 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:45.537445 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 02:28:45.537492 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:28:45.538475 20421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:45.531257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42501 (local address 127.0.98.2:55507)
0619 02:28:45.531856 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:45.531895 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:45.533136 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:45.533624 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:45.533631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:45.533712 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:45.534169 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:45.534181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.534892 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.534894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:45.535847 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:45.535856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.536290 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.536293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:45.536426 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:45.536963 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:28:45.536965 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:45.537003 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:28:45.537061 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:45.537328 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:45.537335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:45.537339 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:28:45.537433 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 02:28:45.537718 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":115,"threads_started":1}
I20260619 02:28:45.539331 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 20291
I20260619 02:28:45.539469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/wal/instance
I20260619 02:28:45.539673 20419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42501
I20260619 02:28:45.539814 392 external_mini_cluster.cc:1428] Setting key 4da92a622f07fd81038b7af1680d815a
I20260619 02:28:45.539903 20419 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:45.540452 20419 heartbeater.cc:511] Master 127.0.98.62:42501 requested a full tablet report, sending...
I20260619 02:28:45.541610 20100 ts_manager.cc:194] Registered new tserver with Master: bc97fe6a6ee14619b138499cc4f19aab (127.0.98.2:46249)
I20260619 02:28:45.541658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42501
--builtin_ntp_servers=127.0.98.20:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:28:45.542176 20100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55507
W20260619 02:28:45.648299 20422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:45.648528 20422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:45.648563 20422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:45.652639 20422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:45.652688 20422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:45.652766 20422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 02:28:45.658113 20422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:45.659071 20422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:45.660367 20422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:45.670322 20427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:45.670322 20430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:45.670341 20428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:45.670426 20422 server_base.cc:1034] running on GCE node
I20260619 02:28:45.670979 20422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:45.671581 20422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:45.672739 20422 hybrid_clock.cc:648] HybridClock initialized: now 1781836125672710 us; error 41 us; skew 500 ppm
I20260619 02:28:45.674485 20422 webserver.cc:460] Webserver started at http://127.0.98.3:38031/ using document root <none> and password file <none>
I20260619 02:28:45.675029 20422 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:45.675092 20422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:45.675354 20422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:45.677380 20422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/data/instance:
uuid: "b857416ffa1f4bdc9ecb61c94f315c28"
format_stamp: "Formatted at 2026-06-19 02:28:45 on dist-test-slave-xkrt"
server_key: "226d6f22fdb0fe61fd3d9daddd335e0e"
server_key_iv: "6163f67a7c56d11e12596eab90c08d3a"
server_key_version: "encryptionkey@0"
I20260619 02:28:45.677981 20422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b857416ffa1f4bdc9ecb61c94f315c28"
format_stamp: "Formatted at 2026-06-19 02:28:45 on dist-test-slave-xkrt"
server_key: "226d6f22fdb0fe61fd3d9daddd335e0e"
server_key_iv: "6163f67a7c56d11e12596eab90c08d3a"
server_key_version: "encryptionkey@0"
I20260619 02:28:45.681689 20422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:28:45.683967 20436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:45.685145 20422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:28:45.685267 20422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/wal
uuid: "b857416ffa1f4bdc9ecb61c94f315c28"
format_stamp: "Formatted at 2026-06-19 02:28:45 on dist-test-slave-xkrt"
server_key: "226d6f22fdb0fe61fd3d9daddd335e0e"
server_key_iv: "6163f67a7c56d11e12596eab90c08d3a"
server_key_version: "encryptionkey@0"
I20260619 02:28:45.685369 20422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:45.705788 20422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:45.706367 20422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:45.706542 20422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:45.707108 20422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:28:45.707998 20422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:28:45.708066 20422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:45.708137 20422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:28:45.708182 20422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:45.716361 20422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36323
I20260619 02:28:45.716389 20549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36323 every 8 connection(s)
I20260619 02:28:45.717224 20422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/data/info.pb
I20260619 02:28:45.725764 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:45.719048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37447 (local address 127.0.98.62:42501)
0619 02:28:45.719212 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:45.719217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:45.720206 (+ 989us) server_negotiation.cc:407] Connection header received
0619 02:28:45.721480 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:45.721483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:45.721528 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:45.721620 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:45.722457 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.722963 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.724029 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.724277 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.724354 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:45.725331 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:45.725334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:45.725336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:45.725347 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:45.725365 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:45.725373 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:45.725435 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:45.725601 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 02:28:45.725653 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:28:45.726526 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:45.719447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42501 (local address 127.0.98.3:37447)
0619 02:28:45.720056 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:45.720094 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:45.721294 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:45.721791 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:45.721797 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:45.721878 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:28:45.722332 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:45.722343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.723063 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.723065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:45.723913 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:45.723922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.724401 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.724404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:45.724533 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:45.725086 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:28:45.725088 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:28:45.725124 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:28:45.725182 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:28:45.725443 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:28:45.725449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:28:45.725453 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:28:45.725582 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 02:28:45.725803 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
I20260619 02:28:45.727270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 20422
I20260619 02:28:45.727376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/wal/instance
I20260619 02:28:45.727679 392 external_mini_cluster.cc:1428] Setting key 08474508d79ad44bd717b787f7197424
I20260619 02:28:45.727705 20550 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42501
I20260619 02:28:45.727924 20550 heartbeater.cc:464] Registering TS with master...
I20260619 02:28:45.728495 20550 heartbeater.cc:511] Master 127.0.98.62:42501 requested a full tablet report, sending...
I20260619 02:28:45.729573 20098 ts_manager.cc:194] Registered new tserver with Master: b857416ffa1f4bdc9ecb61c94f315c28 (127.0.98.3:36323)
I20260619 02:28:45.730172 20098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37447
I20260619 02:28:45.741345 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:28:45.746968 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:45.743265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.0.98.62:42501)
0619 02:28:45.743398 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:45.743403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:45.743587 (+ 184us) server_negotiation.cc:407] Connection header received
0619 02:28:45.743740 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:45.743744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:45.743786 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:45.743872 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:45.744187 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.744687 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.745513 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.745753 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.745860 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:45.746541 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:45.746545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:45.746548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:45.746559 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:45.746582 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:45.746589 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:45.746672 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:45.746779 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 02:28:45.746831 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:28:45.749663 20098 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:28:45.752640 20098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:28:45.766636 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:45.760837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40823 (local address 127.0.0.1:59712)
0619 02:28:45.761500 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:45.761569 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:45.761711 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:45.762321 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:45.762327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:45.762352 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:45.762492 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:45.762499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.763697 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.763701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:45.765027 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:45.765037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.766148 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.766162 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:45.766274 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:45.766309 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 02:28:45.766415 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 02:28:45.766520 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"spinlock_wait_cycles":11904,"thread_start_us":102,"threads_started":1}
I20260619 02:28:45.766997 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:45.760820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.0.98.2:46249)
0619 02:28:45.761171 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:45.761183 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:45.761857 (+ 674us) server_negotiation.cc:407] Connection header received
0619 02:28:45.761943 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:45.761949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:45.762150 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:45.762308 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:45.762758 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.763866 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.765206 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.766015 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.766105 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:45.766297 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:45.766739 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 02:28:45.766852 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":130,"threads_started":1}
I20260619 02:28:45.767445 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:45.761445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.0.98.1:40823)
0619 02:28:45.761884 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:45.761900 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:45.761917 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:28:45.761980 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:45.761985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:45.762150 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:45.762308 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:45.762641 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.763585 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.765190 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.766016 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.766130 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:45.766296 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:45.767193 (+ 897us) server_negotiation.cc:299] Negotiation successful
0619 02:28:45.767282 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":136,"threads_started":1}
I20260619 02:28:45.767441 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:45.762141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.0.98.3:36323)
0619 02:28:45.762759 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:45.762771 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:45.762793 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:28:45.762864 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:45.762869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:45.763041 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:45.763185 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:45.763419 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.764334 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.765571 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.766119 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.766220 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:45.766397 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:45.767180 (+ 783us) server_negotiation.cc:299] Negotiation successful
0619 02:28:45.767283 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":161,"threads_started":1}
I20260619 02:28:45.766954 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:45.761389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36323 (local address 127.0.0.1:34766)
0619 02:28:45.761928 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:45.761940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:45.762013 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:45.763201 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:45.763205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:45.763222 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:45.763305 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:45.763311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.764436 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.764441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:45.765435 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:45.765443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.766376 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.766380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:45.766546 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:45.766615 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 02:28:45.766686 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 02:28:45.766787 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":73,"thread_start_us":83,"threads_started":1}
I20260619 02:28:45.766954 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:45.760683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46249 (local address 127.0.0.1:36064)
0619 02:28:45.761463 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:45.761529 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:45.761712 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:45.762483 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:45.762485 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:45.762499 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:45.762599 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:45.762606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.763978 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.763981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:45.765076 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:45.765086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:45.766380 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:45.766383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:45.766521 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:45.766606 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 02:28:45.766686 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 02:28:45.766791 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":87,"spinlock_wait_cycles":28928,"thread_start_us":81,"threads_started":1}
I20260619 02:28:45.770241 20484 tablet_service.cc:1467] Processing CreateTablet for tablet c150fa74283340f69eb96f3b1b8d6c17 (DEFAULT_TABLE table=test-table [id=5b1736662d2842f29d63869c4939d6ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:45.770236 20353 tablet_service.cc:1467] Processing CreateTablet for tablet c150fa74283340f69eb96f3b1b8d6c17 (DEFAULT_TABLE table=test-table [id=5b1736662d2842f29d63869c4939d6ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:45.771168 20353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c150fa74283340f69eb96f3b1b8d6c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:45.771183 20484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c150fa74283340f69eb96f3b1b8d6c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:45.776938 20568 tablet_bootstrap.cc:492] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28: Bootstrap starting.
I20260619 02:28:45.778249 20567 tablet_bootstrap.cc:492] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab: Bootstrap starting.
I20260619 02:28:45.779500 20568 tablet_bootstrap.cc:654] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:45.779947 20567 tablet_bootstrap.cc:654] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:45.779520 20222 tablet_service.cc:1467] Processing CreateTablet for tablet c150fa74283340f69eb96f3b1b8d6c17 (DEFAULT_TABLE table=test-table [id=5b1736662d2842f29d63869c4939d6ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:28:45.780329 20568 log.cc:824] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:45.780580 20222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c150fa74283340f69eb96f3b1b8d6c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:45.780618 20567 log.cc:824] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:45.782255 20567 tablet_bootstrap.cc:492] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab: No bootstrap required, opened a new log
I20260619 02:28:45.782310 20568 tablet_bootstrap.cc:492] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28: No bootstrap required, opened a new log
I20260619 02:28:45.782451 20567 ts_tablet_manager.cc:1397] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 02:28:45.782481 20568 ts_tablet_manager.cc:1397] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:28:45.785836 20567 raft_consensus.cc:348] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc97fe6a6ee14619b138499cc4f19aab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46249 } } peers { permanent_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40823 } } peers { permanent_uuid: "b857416ffa1f4bdc9ecb61c94f315c28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36323 } }
I20260619 02:28:45.786055 20567 raft_consensus.cc:374] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:45.786114 20567 raft_consensus.cc:729] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc97fe6a6ee14619b138499cc4f19aab, State: Initialized, Role: FOLLOWER
I20260619 02:28:45.786526 20567 consensus_queue.cc:260] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc97fe6a6ee14619b138499cc4f19aab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46249 } } peers { permanent_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40823 } } peers { permanent_uuid: "b857416ffa1f4bdc9ecb61c94f315c28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36323 } }
I20260619 02:28:45.786604 20568 raft_consensus.cc:348] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc97fe6a6ee14619b138499cc4f19aab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46249 } } peers { permanent_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40823 } } peers { permanent_uuid: "b857416ffa1f4bdc9ecb61c94f315c28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36323 } }
I20260619 02:28:45.786887 20568 raft_consensus.cc:374] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:45.786945 20568 raft_consensus.cc:729] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b857416ffa1f4bdc9ecb61c94f315c28, State: Initialized, Role: FOLLOWER
I20260619 02:28:45.787287 20419 heartbeater.cc:503] Master 127.0.98.62:42501 was elected leader, sending a full tablet report...
I20260619 02:28:45.787492 20567 ts_tablet_manager.cc:1428] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 02:28:45.787946 20568 consensus_queue.cc:260] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc97fe6a6ee14619b138499cc4f19aab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46249 } } peers { permanent_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40823 } } peers { permanent_uuid: "b857416ffa1f4bdc9ecb61c94f315c28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36323 } }
I20260619 02:28:45.788283 20572 tablet_bootstrap.cc:492] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c: Bootstrap starting.
I20260619 02:28:45.788879 20568 ts_tablet_manager.cc:1428] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 02:28:45.788887 20550 heartbeater.cc:503] Master 127.0.98.62:42501 was elected leader, sending a full tablet report...
I20260619 02:28:45.790459 20572 tablet_bootstrap.cc:654] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:45.791290 20572 log.cc:824] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:45.793588 20572 tablet_bootstrap.cc:492] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c: No bootstrap required, opened a new log
I20260619 02:28:45.793787 20572 ts_tablet_manager.cc:1397] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 02:28:45.796478 20572 raft_consensus.cc:348] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc97fe6a6ee14619b138499cc4f19aab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46249 } } peers { permanent_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40823 } } peers { permanent_uuid: "b857416ffa1f4bdc9ecb61c94f315c28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36323 } }
I20260619 02:28:45.796669 20572 raft_consensus.cc:374] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:45.796746 20572 raft_consensus.cc:729] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5b53c5b9e54e138db5f87b5dea957c, State: Initialized, Role: FOLLOWER
I20260619 02:28:45.797096 20572 consensus_queue.cc:260] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc97fe6a6ee14619b138499cc4f19aab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46249 } } peers { permanent_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40823 } } peers { permanent_uuid: "b857416ffa1f4bdc9ecb61c94f315c28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36323 } }
I20260619 02:28:45.797672 20288 heartbeater.cc:503] Master 127.0.98.62:42501 was elected leader, sending a full tablet report...
I20260619 02:28:45.797946 20572 ts_tablet_manager.cc:1428] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260619 02:28:45.844441 20289 tablet.cc:2369] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:28:45.968174 20551 tablet.cc:2369] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:28:46.018054 20575 raft_consensus.cc:482] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:28:46.018263 20575 raft_consensus.cc:504] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc97fe6a6ee14619b138499cc4f19aab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46249 } } peers { permanent_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40823 } } peers { permanent_uuid: "b857416ffa1f4bdc9ecb61c94f315c28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36323 } }
I20260619 02:28:46.019409 20575 leader_election.cc:288] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc97fe6a6ee14619b138499cc4f19aab (127.0.98.2:46249), b857416ffa1f4bdc9ecb61c94f315c28 (127.0.98.3:36323)
I20260619 02:28:46.023370 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:46.019803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34213 (local address 127.0.98.3:36323)
0619 02:28:46.019973 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:46.019979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:46.020183 (+ 204us) server_negotiation.cc:407] Connection header received
0619 02:28:46.020336 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:46.020339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:46.020407 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:46.020476 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:46.020752 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:46.021267 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:46.022393 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:46.022792 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:46.022903 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:46.022964 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:46.023162 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 02:28:46.023233 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 02:28:46.023377 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:46.019727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44271 (local address 127.0.98.2:46249)
0619 02:28:46.019875 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:46.019880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:46.020091 (+ 211us) server_negotiation.cc:407] Connection header received
0619 02:28:46.020256 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:46.020262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:46.020322 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:46.020385 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:28:46.020726 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:46.021251 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:46.022430 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:46.022807 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:46.022924 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:46.022982 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:46.023199 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 02:28:46.023272 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:28:46.023399 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:46.019652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36323 (local address 127.0.98.1:34213)
0619 02:28:46.020075 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:46.020090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:46.020224 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:46.020513 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:46.020515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:46.020529 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:46.020625 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:46.020629 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:46.021399 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:46.021402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:46.022265 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:46.022274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:46.022917 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:46.022920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:46.023018 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:46.023054 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:28:46.023112 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 02:28:46.023200 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":37,"thread_start_us":115,"threads_started":1}
I20260619 02:28:46.023399 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:46.019651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46249 (local address 127.0.98.1:44271)
0619 02:28:46.019973 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:46.019988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:28:46.020113 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:28:46.020421 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:28:46.020426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:28:46.020448 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:28:46.020599 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:46.020604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:46.021367 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:46.021371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:46.022276 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:28:46.022286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:28:46.022945 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:28:46.022948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:28:46.023054 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:46.023071 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:28:46.023160 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 02:28:46.023229 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":74,"thread_start_us":85,"threads_started":1}
I20260619 02:28:46.024103 20373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c150fa74283340f69eb96f3b1b8d6c17" candidate_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc97fe6a6ee14619b138499cc4f19aab" is_pre_election: true
I20260619 02:28:46.024106 20504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c150fa74283340f69eb96f3b1b8d6c17" candidate_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b857416ffa1f4bdc9ecb61c94f315c28" is_pre_election: true
I20260619 02:28:46.024394 20504 raft_consensus.cc:2454] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5b53c5b9e54e138db5f87b5dea957c in term 0.
I20260619 02:28:46.024400 20373 raft_consensus.cc:2454] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5b53c5b9e54e138db5f87b5dea957c in term 0.
I20260619 02:28:46.024891 20178 leader_election.cc:302] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3c5b53c5b9e54e138db5f87b5dea957c, b857416ffa1f4bdc9ecb61c94f315c28, bc97fe6a6ee14619b138499cc4f19aab; no voters:
I20260619 02:28:46.025144 20575 raft_consensus.cc:2781] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:28:46.025233 20575 raft_consensus.cc:482] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:28:46.025293 20575 raft_consensus.cc:3037] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:46.026353 20575 raft_consensus.cc:504] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc97fe6a6ee14619b138499cc4f19aab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46249 } } peers { permanent_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40823 } } peers { permanent_uuid: "b857416ffa1f4bdc9ecb61c94f315c28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36323 } }
I20260619 02:28:46.026712 20575 leader_election.cc:288] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [CANDIDATE]: Term 1 election: Requested vote from peers bc97fe6a6ee14619b138499cc4f19aab (127.0.98.2:46249), b857416ffa1f4bdc9ecb61c94f315c28 (127.0.98.3:36323)
I20260619 02:28:46.027066 20373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c150fa74283340f69eb96f3b1b8d6c17" candidate_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc97fe6a6ee14619b138499cc4f19aab"
I20260619 02:28:46.027063 20504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c150fa74283340f69eb96f3b1b8d6c17" candidate_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b857416ffa1f4bdc9ecb61c94f315c28"
I20260619 02:28:46.027187 20504 raft_consensus.cc:3037] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:46.027187 20373 raft_consensus.cc:3037] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:46.028434 20373 raft_consensus.cc:2454] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5b53c5b9e54e138db5f87b5dea957c in term 1.
I20260619 02:28:46.028434 20504 raft_consensus.cc:2454] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5b53c5b9e54e138db5f87b5dea957c in term 1.
I20260619 02:28:46.028746 20177 leader_election.cc:302] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5b53c5b9e54e138db5f87b5dea957c, bc97fe6a6ee14619b138499cc4f19aab; no voters:
I20260619 02:28:46.028939 20575 raft_consensus.cc:2781] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:46.029160 20575 raft_consensus.cc:686] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [term 1 LEADER]: Becoming Leader. State: Replica: 3c5b53c5b9e54e138db5f87b5dea957c, State: Running, Role: LEADER
I20260619 02:28:46.029476 20575 consensus_queue.cc:237] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc97fe6a6ee14619b138499cc4f19aab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46249 } } peers { permanent_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40823 } } peers { permanent_uuid: "b857416ffa1f4bdc9ecb61c94f315c28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36323 } }
W20260619 02:28:46.030200 20420 tablet.cc:2369] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:28:46.032696 20100 catalog_manager.cc:5495] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5b53c5b9e54e138db5f87b5dea957c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc97fe6a6ee14619b138499cc4f19aab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5b53c5b9e54e138db5f87b5dea957c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b857416ffa1f4bdc9ecb61c94f315c28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36323 } health_report { overall_health: UNKNOWN } } }
I20260619 02:28:46.057147 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:46.053647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59722 (local address 127.0.98.1:40823)
0619 02:28:46.053768 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:46.053772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:46.053785 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:28:46.053858 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:46.053860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:46.053933 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:46.054019 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:28:46.054301 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:46.054889 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:46.055809 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:46.056019 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:46.056105 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:46.056411 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:28:46.056848 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:28:46.056940 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:46.056991 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 02:28:46.057037 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:28:46.063144 20504 raft_consensus.cc:1261] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28 [term 1 FOLLOWER]: Refusing update from remote peer 3c5b53c5b9e54e138db5f87b5dea957c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:28:46.063148 20373 raft_consensus.cc:1261] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab [term 1 FOLLOWER]: Refusing update from remote peer 3c5b53c5b9e54e138db5f87b5dea957c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:28:46.063824 20578 consensus_queue.cc:1035] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc97fe6a6ee14619b138499cc4f19aab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:28:46.063997 20575 consensus_queue.cc:1035] T c150fa74283340f69eb96f3b1b8d6c17 P 3c5b53c5b9e54e138db5f87b5dea957c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b857416ffa1f4bdc9ecb61c94f315c28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:28:46.071411 20582 mvcc.cc:205] Tried to move back new op lower bound from 7298400772346765312 to 7298400772219609088. Current Snapshot: MvccSnapshot[applied={T|T < 7298400772346765312}]
I20260619 02:28:46.078580 20100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:28:46.078756 20100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:28:46.081024 20100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c58d563605742169a66da49ff7454f4: Sending DeleteTablet for 3 replicas of tablet c150fa74283340f69eb96f3b1b8d6c17
I20260619 02:28:46.081761 20353 tablet_service.cc:1514] Processing DeleteTablet for tablet c150fa74283340f69eb96f3b1b8d6c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:46 UTC) from {username='slave'} at 127.0.0.1:36064
I20260619 02:28:46.081827 20222 tablet_service.cc:1514] Processing DeleteTablet for tablet c150fa74283340f69eb96f3b1b8d6c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:46 UTC) from {username='slave'} at 127.0.0.1:59712
I20260619 02:28:46.081956 20484 tablet_service.cc:1514] Processing DeleteTablet for tablet c150fa74283340f69eb96f3b1b8d6c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:28:46 UTC) from {username='slave'} at 127.0.0.1:34766
I20260619 02:28:46.082569 20592 tablet_replica.cc:330] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28: stopping tablet replica
I20260619 02:28:46.083036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 20160
I20260619 02:28:46.082892 20592 raft_consensus.cc:2229] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:28:46.084142 20592 raft_consensus.cc:2258] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:28:46.084523 20590 tablet_replica.cc:330] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab: stopping tablet replica
I20260619 02:28:46.084810 20590 raft_consensus.cc:2229] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:28:46.085093 20590 raft_consensus.cc:2258] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:28:46.086166 20592 ts_tablet_manager.cc:1905] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:28:46.086493 20590 ts_tablet_manager.cc:1905] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:28:46.088436 20592 ts_tablet_manager.cc:1918] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:28:46.088548 20592 log.cc:1196] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-2/wal/wals/c150fa74283340f69eb96f3b1b8d6c17
I20260619 02:28:46.088909 20592 ts_tablet_manager.cc:1939] T c150fa74283340f69eb96f3b1b8d6c17 P b857416ffa1f4bdc9ecb61c94f315c28: Deleting consensus metadata
I20260619 02:28:46.089072 20590 ts_tablet_manager.cc:1918] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:28:46.089169 20590 log.cc:1196] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab: Deleting WAL directory at /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781835873883564-392-0/minicluster-data/ts-1/wal/wals/c150fa74283340f69eb96f3b1b8d6c17
I20260619 02:28:46.089440 20590 ts_tablet_manager.cc:1939] T c150fa74283340f69eb96f3b1b8d6c17 P bc97fe6a6ee14619b138499cc4f19aab: Deleting consensus metadata
I20260619 02:28:46.090232 20087 catalog_manager.cc:4841] TS b857416ffa1f4bdc9ecb61c94f315c28 (127.0.98.3:36323): tablet c150fa74283340f69eb96f3b1b8d6c17 (table test-table [id=5b1736662d2842f29d63869c4939d6ca]) successfully deleted
I20260619 02:28:46.090914 20086 catalog_manager.cc:4841] TS bc97fe6a6ee14619b138499cc4f19aab (127.0.98.2:46249): tablet c150fa74283340f69eb96f3b1b8d6c17 (table test-table [id=5b1736662d2842f29d63869c4939d6ca]) successfully deleted
I20260619 02:28:46.095018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 20291
W20260619 02:28:46.095016 20084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40823 (error 108)
I20260619 02:28:46.096369 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:46.096113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40823 (local address 127.0.0.1:59724)
0619 02:28:46.096244 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 02:28:46.096301 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 02:28:46.096513 20084 catalog_manager.cc:4584] TS 3c5b53c5b9e54e138db5f87b5dea957c (127.0.98.1:40823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c150fa74283340f69eb96f3b1b8d6c17: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40823: connect: Connection refused (error 111)
I20260619 02:28:46.101665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 20422
I20260619 02:28:46.106807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 20069
2026-06-19T02:28:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1224 ms)
[----------] 6 tests from EncryptionPolicyTest (11091 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:28:46 dist-test-slave-xkrt krb5kdc[20596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:28:46 dist-test-slave-xkrt krb5kdc[20596](info): set up 2 sockets
Jun 19 02:28:46 dist-test-slave-xkrt krb5kdc[20596](info): commencing operation
krb5kdc: starting...
W20260619 02:28:47.153877 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:28:47 dist-test-slave-xkrt krb5kdc[20596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:28:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:28:47.272693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:47.380648 20609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:47.380931 20609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:47.381017 20609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:47.384963 20609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:47.385051 20609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:47.385093 20609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:47.385147 20609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:47.385185 20609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:47.390612 20609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41979
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:47.391633 20609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:47.392633 20609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:47.400705 20617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:47.400876 20609 server_base.cc:1034] running on GCE node
W20260619 02:28:47.400733 20614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:47.401024 20615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:47.401620 20609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:47.402704 20609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:47.403885 20609 hybrid_clock.cc:648] HybridClock initialized: now 1781836127403873 us; error 25 us; skew 500 ppm
Jun 19 02:28:47 dist-test-slave-xkrt krb5kdc[20596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:47.406636 20609 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:28:47.407410 20609 webserver.cc:460] Webserver started at http://127.0.0.1:45927/ using document root <none> and password file <none>
I20260619 02:28:47.407896 20609 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:47.407949 20609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:47.408097 20609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:47.410043 20609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "55d32494e6a446859f665e0892d97352"
format_stamp: "Formatted at 2026-06-19 02:28:47 on dist-test-slave-xkrt"
server_key: "5c01c526d61f6758f2440dcbcfabf2ef"
server_key_iv: "164354f3b3d2eb87261134bb3e91b033"
server_key_version: "encryptionkey@0"
I20260619 02:28:47.410526 20609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "55d32494e6a446859f665e0892d97352"
format_stamp: "Formatted at 2026-06-19 02:28:47 on dist-test-slave-xkrt"
server_key: "5c01c526d61f6758f2440dcbcfabf2ef"
server_key_iv: "164354f3b3d2eb87261134bb3e91b033"
server_key_version: "encryptionkey@0"
I20260619 02:28:47.413865 20609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:28:47.416011 20624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:47.417131 20609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:28:47.417273 20609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "55d32494e6a446859f665e0892d97352"
format_stamp: "Formatted at 2026-06-19 02:28:47 on dist-test-slave-xkrt"
server_key: "5c01c526d61f6758f2440dcbcfabf2ef"
server_key_iv: "164354f3b3d2eb87261134bb3e91b033"
server_key_version: "encryptionkey@0"
I20260619 02:28:47.417387 20609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:47.432440 20609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:47.435825 20609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:47.436017 20609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:47.442246 20609 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41979
I20260619 02:28:47.442302 20676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41979 every 8 connection(s)
I20260619 02:28:47.443146 20609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:47.445815 20677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:47.447885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 20609
I20260619 02:28:47.448028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:47.448318 392 external_mini_cluster.cc:1428] Setting key 762bef0cfc354d72d86e27e1e581d8c5
I20260619 02:28:47.451373 20677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352: Bootstrap starting.
I20260619 02:28:47.453799 20677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:47.454607 20677 log.cc:824] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:47.456748 20677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352: No bootstrap required, opened a new log
Jun 19 02:28:47 dist-test-slave-xkrt krb5kdc[20596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:28:47.461320 20677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d32494e6a446859f665e0892d97352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } }
I20260619 02:28:47.461611 20677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:47.461673 20677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d32494e6a446859f665e0892d97352, State: Initialized, Role: FOLLOWER
I20260619 02:28:47.462126 20677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d32494e6a446859f665e0892d97352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } }
I20260619 02:28:47.462289 20677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:47.462350 20677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:47.462440 20677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:47.462764 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:47.449718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57495 (local address 127.0.0.1:41979)
0619 02:28:47.450090 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:47.450105 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:47.450144 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:28:47.450748 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:47.450763 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:47.451076 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:47.451351 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:47.452229 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:47.453174 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:47.454341 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:47.454844 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:47.454991 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:47.457738 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:47.457744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:47.457758 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:47.457787 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:47.459724 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:47.460182 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:47.460188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:47.460193 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:47.460324 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:47.460610 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:47.460613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:47.460615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:47.461093 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:47.461303 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:47.461502 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 02:28:47.461802 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":119,"threads_started":1}
I20260619 02:28:47.463371 20677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d32494e6a446859f665e0892d97352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } }
I20260619 02:28:47.463714 20677 leader_election.cc:302] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55d32494e6a446859f665e0892d97352; no voters:
I20260619 02:28:47.464039 20677 leader_election.cc:288] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:47.464161 20682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:47.464385 20682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [term 1 LEADER]: Becoming Leader. State: Replica: 55d32494e6a446859f665e0892d97352, State: Running, Role: LEADER
I20260619 02:28:47.464675 20682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d32494e6a446859f665e0892d97352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } }
I20260619 02:28:47.465237 20677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:47.465703 20682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55d32494e6a446859f665e0892d97352. Latest consensus state: current_term: 1 leader_uuid: "55d32494e6a446859f665e0892d97352" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d32494e6a446859f665e0892d97352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } } }
I20260619 02:28:47.465788 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:28:47.465823 20682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:47.466236 20687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:47.466221 20684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55d32494e6a446859f665e0892d97352" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d32494e6a446859f665e0892d97352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } } }
I20260619 02:28:47.466430 20684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:47.469014 20687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:47.474447 20687 catalog_manager.cc:1347] Generated new cluster ID: 6af4391b0739419ba85f4ac77769dcb0
I20260619 02:28:47.474522 20687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:47.477661 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:47.468090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41807 (local address 127.0.0.1:41979)
0619 02:28:47.468191 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:47.468194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:47.469808 (+ 1614us) server_negotiation.cc:407] Connection header received
0619 02:28:47.469921 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:47.469923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:47.469957 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:47.470047 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:47.470714 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:47.471428 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:47.472569 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:47.472901 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:47.473014 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:47.474420 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:47.474424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:47.474427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:47.474449 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:47.475969 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:47.476420 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:47.476424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:47.476425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:47.476504 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:47.476812 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:47.476815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:47.476816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:47.476978 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:47.477134 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:47.477380 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 02:28:47.477467 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:28:47.493067 20687 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:47.494107 20687 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:47.500121 20687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55d32494e6a446859f665e0892d97352: Generated new TSK 0
I20260619 02:28:47.500862 20687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:28:47.513903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 20609
W20260619 02:28:47.519434 20594 connection.cc:535] client connection to 127.0.0.1:41979 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41979): Connection reset by peer (error 104)
2026-06-19T02:28:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:28:47 dist-test-slave-xkrt krb5kdc[20708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:28:47 dist-test-slave-xkrt krb5kdc[20708](info): set up 2 sockets
Jun 19 02:28:47 dist-test-slave-xkrt krb5kdc[20708](info): commencing operation
krb5kdc: starting...
W20260619 02:28:48.561800 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:28:48 dist-test-slave-xkrt krb5kdc[20708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:28:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:28:48.677205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36915
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:48.782727 20721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:48.782970 20721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:48.783010 20721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:48.787045 20721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:48.787098 20721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:48.787118 20721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:48.787134 20721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:48.787189 20721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:48.792253 20721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36915
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:48.793213 20721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:48.794185 20721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:48.802268 20727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:48.802367 20721 server_base.cc:1034] running on GCE node
W20260619 02:28:48.802276 20726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:48.802284 20729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:48.803025 20721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:48.804118 20721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:48.805306 20721 hybrid_clock.cc:648] HybridClock initialized: now 1781836128805295 us; error 29 us; skew 500 ppm
Jun 19 02:28:48 dist-test-slave-xkrt krb5kdc[20708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:48.808108 20721 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:28:48.808892 20721 webserver.cc:460] Webserver started at http://127.0.0.1:39499/ using document root <none> and password file <none>
I20260619 02:28:48.809377 20721 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:48.809434 20721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:48.809669 20721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:48.811595 20721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "7abd3ba7ba404d3b980b1d9b4fe3e646"
format_stamp: "Formatted at 2026-06-19 02:28:48 on dist-test-slave-xkrt"
server_key: "a61fd63a38d2c85004c10f9a263b91bb"
server_key_iv: "5ff94322b710d5c9bc3c2b04ea7d8812"
server_key_version: "encryptionkey@0"
I20260619 02:28:48.812343 20721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "7abd3ba7ba404d3b980b1d9b4fe3e646"
format_stamp: "Formatted at 2026-06-19 02:28:48 on dist-test-slave-xkrt"
server_key: "a61fd63a38d2c85004c10f9a263b91bb"
server_key_iv: "5ff94322b710d5c9bc3c2b04ea7d8812"
server_key_version: "encryptionkey@0"
I20260619 02:28:48.815928 20721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 02:28:48.818167 20736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:48.819559 20721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:28:48.819657 20721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "7abd3ba7ba404d3b980b1d9b4fe3e646"
format_stamp: "Formatted at 2026-06-19 02:28:48 on dist-test-slave-xkrt"
server_key: "a61fd63a38d2c85004c10f9a263b91bb"
server_key_iv: "5ff94322b710d5c9bc3c2b04ea7d8812"
server_key_version: "encryptionkey@0"
I20260619 02:28:48.819748 20721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:48.833465 20721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:48.836571 20721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:48.836719 20721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:48.843250 20721 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36915
I20260619 02:28:48.843273 20788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36915 every 8 connection(s)
I20260619 02:28:48.844216 20721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:48.846793 20789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:48.851963 20789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646: Bootstrap starting.
I20260619 02:28:48.852618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 20721
I20260619 02:28:48.852758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:48.853021 392 external_mini_cluster.cc:1428] Setting key 8c35fc1012f8e27a2eeb25b00c11bb91
I20260619 02:28:48.854632 20789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:48.855341 20789 log.cc:824] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:48.857053 20789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646: No bootstrap required, opened a new log
I20260619 02:28:48.860459 20789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7abd3ba7ba404d3b980b1d9b4fe3e646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36915 } }
I20260619 02:28:48.860723 20789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:48.860790 20789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abd3ba7ba404d3b980b1d9b4fe3e646, State: Initialized, Role: FOLLOWER
I20260619 02:28:48.861217 20789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7abd3ba7ba404d3b980b1d9b4fe3e646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36915 } }
I20260619 02:28:48.861335 20789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:48.861415 20789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:48.861513 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:48.854181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58077 (local address 127.0.0.1:36915)
0619 02:28:48.854579 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:48.854594 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:48.854628 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:28:48.855276 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:48.855292 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:48.855652 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:48.855972 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:48.856363 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:48.857329 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:48.858249 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:48.858787 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:48.858943 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:48.859822 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:48.859828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:48.859848 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:48.860210 (+ 362us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:48.860261 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:48.860283 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:48.860415 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:48.860564 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 02:28:48.860715 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":127,"threads_started":1}
I20260619 02:28:48.861517 20789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:48.862427 20789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7abd3ba7ba404d3b980b1d9b4fe3e646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36915 } }
I20260619 02:28:48.862746 20789 leader_election.cc:302] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7abd3ba7ba404d3b980b1d9b4fe3e646; no voters:
I20260619 02:28:48.863019 20789 leader_election.cc:288] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:48.863099 20794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:48.863353 20794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [term 1 LEADER]: Becoming Leader. State: Replica: 7abd3ba7ba404d3b980b1d9b4fe3e646, State: Running, Role: LEADER
I20260619 02:28:48.863632 20794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7abd3ba7ba404d3b980b1d9b4fe3e646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36915 } }
I20260619 02:28:48.864243 20789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:48.864322 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:28:48.867036 20795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7abd3ba7ba404d3b980b1d9b4fe3e646. Latest consensus state: current_term: 1 leader_uuid: "7abd3ba7ba404d3b980b1d9b4fe3e646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7abd3ba7ba404d3b980b1d9b4fe3e646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36915 } } }
I20260619 02:28:48.867183 20795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:48.867451 20796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7abd3ba7ba404d3b980b1d9b4fe3e646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7abd3ba7ba404d3b980b1d9b4fe3e646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36915 } } }
I20260619 02:28:48.867552 20796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646 [sys.catalog]: This master's current role is: LEADER
W20260619 02:28:48.869347 20813 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:28:48.869419 20813 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:28:48.869510 20815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:48.870368 20815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:48.871232 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:48.866679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57093 (local address 127.0.0.1:36915)
0619 02:28:48.866802 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:48.866808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:48.867140 (+ 332us) server_negotiation.cc:407] Connection header received
0619 02:28:48.867290 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:48.867296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:48.867349 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:48.867406 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:48.867813 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:48.868519 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:48.869656 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:48.869883 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:48.869992 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:48.870782 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:48.870785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:48.870788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:48.870799 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:48.870824 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:48.870840 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:48.870930 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:48.871047 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 02:28:48.871079 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:28:48.875286 20815 catalog_manager.cc:1347] Generated new cluster ID: d8273bda0ac34f1cbd4c2bb3c0f289de
I20260619 02:28:48.875346 20815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:48.891196 20815 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:48.892647 20815 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:48.895831 20815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7abd3ba7ba404d3b980b1d9b4fe3e646: Generated new TSK 0
I20260619 02:28:48.896435 20815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:28:48.901186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 20721
W20260619 02:28:48.906474 20706 connection.cc:535] client connection to 127.0.0.1:36915 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36915): Connection reset by peer (error 104)
2026-06-19T02:28:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:28:48 dist-test-slave-xkrt krb5kdc[20820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:28:48 dist-test-slave-xkrt krb5kdc[20820](info): set up 2 sockets
Jun 19 02:28:48 dist-test-slave-xkrt krb5kdc[20820](info): commencing operation
krb5kdc: starting...
W20260619 02:28:49.937665 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:28:50 dist-test-slave-xkrt krb5kdc[20820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:28:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:28:50.056481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:50.163861 20833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:50.164119 20833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:50.164192 20833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:50.168141 20833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:50.168198 20833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:50.168217 20833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:50.168233 20833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:50.168274 20833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:50.173486 20833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:50.174468 20833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:50.175432 20833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:50.183523 20841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:50.183530 20839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:50.183529 20838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:50.183760 20833 server_base.cc:1034] running on GCE node
I20260619 02:28:50.184368 20833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:50.185397 20833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:50.186584 20833 hybrid_clock.cc:648] HybridClock initialized: now 1781836130186554 us; error 49 us; skew 500 ppm
Jun 19 02:28:50 dist-test-slave-xkrt krb5kdc[20820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:50.189200 20833 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:28:50.190079 20833 webserver.cc:460] Webserver started at http://127.0.0.1:39323/ using document root <none> and password file <none>
I20260619 02:28:50.190615 20833 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:50.190702 20833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:50.190915 20833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:50.192835 20833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "4a7c657dd25d4ea593e12f19f83a574b"
format_stamp: "Formatted at 2026-06-19 02:28:50 on dist-test-slave-xkrt"
server_key: "d8c438470053fcfbfcb1f855628484e3"
server_key_iv: "d9d63192b4acbbc0ccd3d7688f1679b5"
server_key_version: "encryptionkey@0"
I20260619 02:28:50.193370 20833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a7c657dd25d4ea593e12f19f83a574b"
format_stamp: "Formatted at 2026-06-19 02:28:50 on dist-test-slave-xkrt"
server_key: "d8c438470053fcfbfcb1f855628484e3"
server_key_iv: "d9d63192b4acbbc0ccd3d7688f1679b5"
server_key_version: "encryptionkey@0"
I20260619 02:28:50.196913 20833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:28:50.199162 20848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:50.200330 20833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:28:50.200469 20833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "4a7c657dd25d4ea593e12f19f83a574b"
format_stamp: "Formatted at 2026-06-19 02:28:50 on dist-test-slave-xkrt"
server_key: "d8c438470053fcfbfcb1f855628484e3"
server_key_iv: "d9d63192b4acbbc0ccd3d7688f1679b5"
server_key_version: "encryptionkey@0"
I20260619 02:28:50.200574 20833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:50.204820 20833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:50.207437 20833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:50.207617 20833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:50.214008 20833 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45341
I20260619 02:28:50.214059 20900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45341 every 8 connection(s)
I20260619 02:28:50.214869 20833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:50.217432 20901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:50.221663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 20833
I20260619 02:28:50.221830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:50.222131 392 external_mini_cluster.cc:1428] Setting key f2ee126d2a79d6d1d69bd27f48aeaec9
I20260619 02:28:50.223088 20901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b: Bootstrap starting.
I20260619 02:28:50.225342 20901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:50.226099 20901 log.cc:824] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:50.226912 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:50.223373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37925 (local address 127.0.0.1:45341)
0619 02:28:50.223785 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:50.223798 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:50.223831 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:28:50.224449 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:50.224465 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:50.224733 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:50.225049 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:50.225793 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:50.225799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:50.225817 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:50.225827 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:50.225876 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:50.225895 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:50.226000 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:50.226121 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 02:28:50.226269 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":126,"threads_started":1}
I20260619 02:28:50.227989 20901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b: No bootstrap required, opened a new log
I20260619 02:28:50.229601 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:28:50.232131 20901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7c657dd25d4ea593e12f19f83a574b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45341 } }
I20260619 02:28:50.232412 20901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:50.232474 20901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7c657dd25d4ea593e12f19f83a574b, State: Initialized, Role: FOLLOWER
I20260619 02:28:50.232901 20901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7c657dd25d4ea593e12f19f83a574b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45341 } }
I20260619 02:28:50.233039 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:50.231442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44757 (local address 127.0.0.1:45341)
0619 02:28:50.231541 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:50.231544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:50.231723 (+ 179us) server_negotiation.cc:407] Connection header received
0619 02:28:50.231840 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:50.231842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:50.231870 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:50.231932 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:50.232631 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:50.232634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:50.232636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:50.232652 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:50.232674 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:50.232684 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:50.232770 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:50.232839 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 02:28:50.232874 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:28:50.233107 20901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:50.233186 20901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:50.233287 20901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:50.234356 20901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7c657dd25d4ea593e12f19f83a574b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45341 } }
I20260619 02:28:50.234680 20901 leader_election.cc:302] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a7c657dd25d4ea593e12f19f83a574b; no voters:
I20260619 02:28:50.234956 20901 leader_election.cc:288] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:50.235049 20911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:50.235289 20911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [term 1 LEADER]: Becoming Leader. State: Replica: 4a7c657dd25d4ea593e12f19f83a574b, State: Running, Role: LEADER
I20260619 02:28:50.235576 20911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7c657dd25d4ea593e12f19f83a574b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45341 } }
I20260619 02:28:50.236140 20901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:50.237145 20913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a7c657dd25d4ea593e12f19f83a574b. Latest consensus state: current_term: 1 leader_uuid: "4a7c657dd25d4ea593e12f19f83a574b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7c657dd25d4ea593e12f19f83a574b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45341 } } }
I20260619 02:28:50.237293 20913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:50.237576 20912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a7c657dd25d4ea593e12f19f83a574b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7c657dd25d4ea593e12f19f83a574b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45341 } } }
I20260619 02:28:50.237676 20912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:50.237766 20920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:50.240418 20920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:50.245571 20920 catalog_manager.cc:1347] Generated new cluster ID: ab27d072768944e489f43b555c2d8600
I20260619 02:28:50.245637 20920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:50.258734 20920 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:50.259021 20920 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:50.274132 20920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a7c657dd25d4ea593e12f19f83a574b: Generated new TSK 0
I20260619 02:28:50.274885 20920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:28:50.289038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 20833
2026-06-19T02:28:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:28:50 dist-test-slave-xkrt krb5kdc[20932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:28:50 dist-test-slave-xkrt krb5kdc[20932](info): set up 2 sockets
Jun 19 02:28:50 dist-test-slave-xkrt krb5kdc[20932](info): commencing operation
krb5kdc: starting...
W20260619 02:28:51.337659 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:28:51 dist-test-slave-xkrt krb5kdc[20932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:28:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:28:51.454761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:51.563027 20945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:51.563270 20945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:51.563336 20945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:51.567430 20945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:51.567481 20945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:51.567500 20945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:51.567516 20945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:51.567570 20945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:51.572677 20945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:51.573670 20945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:51.574608 20945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:51.582880 20951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:51.582993 20953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:51.582996 20945 server_base.cc:1034] running on GCE node
W20260619 02:28:51.583199 20950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:51.583765 20945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:51.584818 20945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:51.586035 20945 hybrid_clock.cc:648] HybridClock initialized: now 1781836131586016 us; error 32 us; skew 500 ppm
Jun 19 02:28:51 dist-test-slave-xkrt krb5kdc[20932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:51.588791 20945 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:28:51.589773 20945 webserver.cc:460] Webserver started at http://127.0.0.1:35587/ using document root <none> and password file <none>
I20260619 02:28:51.590314 20945 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:51.590402 20945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:51.590626 20945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:51.592553 20945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "d6953189ccf6460c8869dfba0d70e97e"
format_stamp: "Formatted at 2026-06-19 02:28:51 on dist-test-slave-xkrt"
server_key: "76a269c0c94389c4887db246897beb3e"
server_key_iv: "5fd37e65eb1bb6451089b78e52d65231"
server_key_version: "encryptionkey@0"
I20260619 02:28:51.593098 20945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "d6953189ccf6460c8869dfba0d70e97e"
format_stamp: "Formatted at 2026-06-19 02:28:51 on dist-test-slave-xkrt"
server_key: "76a269c0c94389c4887db246897beb3e"
server_key_iv: "5fd37e65eb1bb6451089b78e52d65231"
server_key_version: "encryptionkey@0"
I20260619 02:28:51.596618 20945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:28:51.598927 20960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:51.600157 20945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:28:51.600293 20945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "d6953189ccf6460c8869dfba0d70e97e"
format_stamp: "Formatted at 2026-06-19 02:28:51 on dist-test-slave-xkrt"
server_key: "76a269c0c94389c4887db246897beb3e"
server_key_iv: "5fd37e65eb1bb6451089b78e52d65231"
server_key_version: "encryptionkey@0"
I20260619 02:28:51.600400 20945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:51.628539 20945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:51.631506 20945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:51.631711 20945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:51.638499 20945 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40037
I20260619 02:28:51.638513 21012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40037 every 8 connection(s)
I20260619 02:28:51.639444 20945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:51.640209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 20945
I20260619 02:28:51.640358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:51.640658 392 external_mini_cluster.cc:1428] Setting key 5c8843eae369a3eea257986ca351c114
I20260619 02:28:51.642876 21013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:51.648383 21013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e: Bootstrap starting.
Jun 19 02:28:51 dist-test-slave-xkrt krb5kdc[20932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:28:51.650990 21013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:51.651733 21013 log.cc:824] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:51.653961 21013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e: No bootstrap required, opened a new log
I20260619 02:28:51.654861 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:51.642047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49043 (local address 127.0.0.1:40037)
0619 02:28:51.642483 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:51.642496 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:51.642530 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:28:51.643219 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:51.643236 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:51.643552 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:51.643886 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:51.644684 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:51.645727 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:51.646563 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:51.647021 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:51.647196 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:51.650022 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:51.650026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:51.650038 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:51.650063 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:51.651772 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:51.652253 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:51.652258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:51.652262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:51.652370 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:51.652664 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:51.652667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:51.652669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:51.653147 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:51.653338 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:51.653605 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 02:28:51.653903 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":148,"threads_started":1}
I20260619 02:28:51.657784 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:28:51.658406 21013 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6953189ccf6460c8869dfba0d70e97e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40037 } }
I20260619 02:28:51.658639 21013 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:51.658735 21013 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6953189ccf6460c8869dfba0d70e97e, State: Initialized, Role: FOLLOWER
I20260619 02:28:51.659209 21013 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6953189ccf6460c8869dfba0d70e97e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40037 } }
I20260619 02:28:51.659366 21013 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:51.659461 21013 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:51.659575 21013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:51.660558 21013 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6953189ccf6460c8869dfba0d70e97e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40037 } }
I20260619 02:28:51.660845 21013 leader_election.cc:302] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6953189ccf6460c8869dfba0d70e97e; no voters:
I20260619 02:28:51.661123 21013 leader_election.cc:288] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:51.661199 21023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:51.661396 21023 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [term 1 LEADER]: Becoming Leader. State: Replica: d6953189ccf6460c8869dfba0d70e97e, State: Running, Role: LEADER
I20260619 02:28:51.661737 21023 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6953189ccf6460c8869dfba0d70e97e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40037 } }
I20260619 02:28:51.662418 21013 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:51.664060 21025 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6953189ccf6460c8869dfba0d70e97e. Latest consensus state: current_term: 1 leader_uuid: "d6953189ccf6460c8869dfba0d70e97e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6953189ccf6460c8869dfba0d70e97e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40037 } } }
I20260619 02:28:51.664431 21025 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:51.664320 21024 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6953189ccf6460c8869dfba0d70e97e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6953189ccf6460c8869dfba0d70e97e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40037 } } }
I20260619 02:28:51.664937 21024 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e [sys.catalog]: This master's current role is: LEADER
W20260619 02:28:51.667896 21039 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:28:51.667991 21039 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:28:51.668083 21038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:51.668632 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:51.659830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56813 (local address 127.0.0.1:40037)
0619 02:28:51.659940 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:51.659944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:51.660101 (+ 157us) server_negotiation.cc:407] Connection header received
0619 02:28:51.660250 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:51.660253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:51.660292 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:51.660381 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:51.661097 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:51.661833 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:51.662720 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:51.663048 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:51.663167 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:51.664944 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:51.664948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:51.664951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:51.664974 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:51.666655 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:51.667113 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:51.667117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:51.667119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:51.667208 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:51.667500 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:51.667519 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:51.667521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:51.667728 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:51.667859 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:51.668365 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 02:28:51.668471 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:28:51.668931 21038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:51.673771 21038 catalog_manager.cc:1347] Generated new cluster ID: a023d8a2db0c461b9f2dbfc6877d588d
I20260619 02:28:51.673831 21038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:51.692559 21038 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:51.693884 21038 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:51.709299 21038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6953189ccf6460c8869dfba0d70e97e: Generated new TSK 0
I20260619 02:28:51.710024 21038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:28:51.720652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 20945
2026-06-19T02:28:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:28:51 dist-test-slave-xkrt krb5kdc[21044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:28:51 dist-test-slave-xkrt krb5kdc[21044](info): set up 2 sockets
Jun 19 02:28:51 dist-test-slave-xkrt krb5kdc[21044](info): commencing operation
krb5kdc: starting...
W20260619 02:28:52.749825 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:28:52 dist-test-slave-xkrt krb5kdc[21044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:28:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:28:52.864645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:52.969058 21057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:52.969329 21057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:52.969408 21057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:52.973294 21057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:52.973376 21057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:52.973414 21057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:52.973470 21057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:52.973506 21057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:52.978564 21057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:52.979512 21057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:52.980448 21057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:52.988580 21065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:52.988590 21062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:52.988590 21063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:52.988664 21057 server_base.cc:1034] running on GCE node
I20260619 02:28:52.989292 21057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:52.990366 21057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:52.991532 21057 hybrid_clock.cc:648] HybridClock initialized: now 1781836132991519 us; error 25 us; skew 500 ppm
Jun 19 02:28:52 dist-test-slave-xkrt krb5kdc[21044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:52.994217 21057 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:28:52.995023 21057 webserver.cc:460] Webserver started at http://127.0.0.1:34565/ using document root <none> and password file <none>
I20260619 02:28:52.995507 21057 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:52.995563 21057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:52.995723 21057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:52.997635 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "85723c05f0db427c9d775f708c6f3570"
format_stamp: "Formatted at 2026-06-19 02:28:52 on dist-test-slave-xkrt"
server_key: "6937be16fd146b292f292bd95ac699a0"
server_key_iv: "be0c51f8df88e6dc5603a9f5ad0a075b"
server_key_version: "encryptionkey@0"
I20260619 02:28:52.998142 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "85723c05f0db427c9d775f708c6f3570"
format_stamp: "Formatted at 2026-06-19 02:28:52 on dist-test-slave-xkrt"
server_key: "6937be16fd146b292f292bd95ac699a0"
server_key_iv: "be0c51f8df88e6dc5603a9f5ad0a075b"
server_key_version: "encryptionkey@0"
I20260619 02:28:53.001452 21057 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260619 02:28:53.003693 21072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:53.004848 21057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:28:53.004966 21057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "85723c05f0db427c9d775f708c6f3570"
format_stamp: "Formatted at 2026-06-19 02:28:52 on dist-test-slave-xkrt"
server_key: "6937be16fd146b292f292bd95ac699a0"
server_key_iv: "be0c51f8df88e6dc5603a9f5ad0a075b"
server_key_version: "encryptionkey@0"
I20260619 02:28:53.005061 21057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:53.025108 21057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:53.028237 21057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:53.028427 21057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:53.035465 21057 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37859
I20260619 02:28:53.035491 21124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37859 every 8 connection(s)
I20260619 02:28:53.036350 21057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:53.039000 21125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:53.040046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 21057
I20260619 02:28:53.040207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:53.040509 392 external_mini_cluster.cc:1428] Setting key 431d943cd73e4103050301f370ecb38a
I20260619 02:28:53.044934 21125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570: Bootstrap starting.
I20260619 02:28:53.047086 21125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:53.047797 21125 log.cc:824] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:53.048722 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:53.041852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38473 (local address 127.0.0.1:37859)
0619 02:28:53.042258 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:53.042272 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:53.042306 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:28:53.042905 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:53.042920 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:53.043245 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:53.043561 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:53.043957 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:53.044868 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:53.045830 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:53.046287 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:53.046455 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:53.047244 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:53.047249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:53.047261 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:53.047611 (+ 350us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:53.047658 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:53.047679 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:53.047782 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:53.047891 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 02:28:53.048079 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":129,"threads_started":1}
I20260619 02:28:53.049714 21125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570: No bootstrap required, opened a new log
I20260619 02:28:53.051383 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:28:53.053776 21125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85723c05f0db427c9d775f708c6f3570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260619 02:28:53.054006 21125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:53.054064 21125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85723c05f0db427c9d775f708c6f3570, State: Initialized, Role: FOLLOWER
I20260619 02:28:53.054445 21125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85723c05f0db427c9d775f708c6f3570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260619 02:28:53.054592 21125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:53.054685 21125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:53.054801 21125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:53.055858 21125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85723c05f0db427c9d775f708c6f3570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260619 02:28:53.056195 21125 leader_election.cc:302] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85723c05f0db427c9d775f708c6f3570; no voters:
I20260619 02:28:53.056526 21125 leader_election.cc:288] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:53.056593 21135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:53.056795 21135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [term 1 LEADER]: Becoming Leader. State: Replica: 85723c05f0db427c9d775f708c6f3570, State: Running, Role: LEADER
I20260619 02:28:53.057142 21135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85723c05f0db427c9d775f708c6f3570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260619 02:28:53.057634 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:53.053367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59505 (local address 127.0.0.1:37859)
0619 02:28:53.053484 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:53.053491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:53.053722 (+ 231us) server_negotiation.cc:407] Connection header received
0619 02:28:53.053831 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:53.053834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:53.053878 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:53.053957 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:53.054193 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:53.054849 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:53.056000 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:53.056257 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:53.056369 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:53.057232 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:53.057237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:53.057241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:53.057255 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:53.057281 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:53.057295 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:53.057358 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:53.057444 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 02:28:53.057489 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:28:53.057812 21125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:53.059445 21137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85723c05f0db427c9d775f708c6f3570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85723c05f0db427c9d775f708c6f3570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } } }
I20260619 02:28:53.059450 21136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85723c05f0db427c9d775f708c6f3570. Latest consensus state: current_term: 1 leader_uuid: "85723c05f0db427c9d775f708c6f3570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85723c05f0db427c9d775f708c6f3570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } } }
I20260619 02:28:53.059587 21137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:53.059594 21136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570 [sys.catalog]: This master's current role is: LEADER
W20260619 02:28:53.062304 21149 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:28:53.062392 21149 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:28:53.062497 21150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:53.063210 21150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:53.068502 21150 catalog_manager.cc:1347] Generated new cluster ID: be45c494063745ac85aa028c2a9ea90d
I20260619 02:28:53.068602 21150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:53.084587 21150 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:53.085820 21150 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:53.095418 21150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85723c05f0db427c9d775f708c6f3570: Generated new TSK 0
I20260619 02:28:53.096103 21150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:28:53.105857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 21057
2026-06-19T02:28:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:28:53 dist-test-slave-xkrt krb5kdc[21156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:28:53 dist-test-slave-xkrt krb5kdc[21156](info): set up 2 sockets
Jun 19 02:28:53 dist-test-slave-xkrt krb5kdc[21156](info): commencing operation
krb5kdc: starting...
W20260619 02:28:54.141767 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:28:54 dist-test-slave-xkrt krb5kdc[21156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:28:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:28:54.260469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:54.366914 21169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:54.367195 21169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:54.367283 21169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:54.371248 21169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:54.371321 21169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:54.371366 21169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:54.371407 21169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:54.371441 21169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:54.376628 21169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:54.377660 21169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:54.378618 21169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:54.387146 21177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:54.387192 21174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:54.387192 21175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:54.387264 21169 server_base.cc:1034] running on GCE node
I20260619 02:28:54.387915 21169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:54.388973 21169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:54.390128 21169 hybrid_clock.cc:648] HybridClock initialized: now 1781836134390105 us; error 36 us; skew 500 ppm
Jun 19 02:28:54 dist-test-slave-xkrt krb5kdc[21156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:54.392829 21169 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:28:54.393735 21169 webserver.cc:460] Webserver started at http://127.0.0.1:39929/ using document root <none> and password file <none>
I20260619 02:28:54.394275 21169 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:54.394333 21169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:54.394559 21169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:54.396485 21169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "de134e913b974fff813424526ba8f7fd"
format_stamp: "Formatted at 2026-06-19 02:28:54 on dist-test-slave-xkrt"
server_key: "eba9ade4ba18c7fa0e4e217d7c124443"
server_key_iv: "f15c8a32304d65f8328e0714d0753dbc"
server_key_version: "encryptionkey@0"
I20260619 02:28:54.397028 21169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "de134e913b974fff813424526ba8f7fd"
format_stamp: "Formatted at 2026-06-19 02:28:54 on dist-test-slave-xkrt"
server_key: "eba9ade4ba18c7fa0e4e217d7c124443"
server_key_iv: "f15c8a32304d65f8328e0714d0753dbc"
server_key_version: "encryptionkey@0"
I20260619 02:28:54.400525 21169 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:28:54.402858 21184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:54.404102 21169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:28:54.404251 21169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "de134e913b974fff813424526ba8f7fd"
format_stamp: "Formatted at 2026-06-19 02:28:54 on dist-test-slave-xkrt"
server_key: "eba9ade4ba18c7fa0e4e217d7c124443"
server_key_iv: "f15c8a32304d65f8328e0714d0753dbc"
server_key_version: "encryptionkey@0"
I20260619 02:28:54.404362 21169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:54.408749 21169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:54.411836 21169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:54.412032 21169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:54.418538 21169 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33003
I20260619 02:28:54.418640 21236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33003 every 8 connection(s)
I20260619 02:28:54.419435 21169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:54.422166 21237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:54.425673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 21169
I20260619 02:28:54.425834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:54.426090 392 external_mini_cluster.cc:1428] Setting key c18387ce9032edd024640b5756386e69
I20260619 02:28:54.427685 21237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd: Bootstrap starting.
I20260619 02:28:54.430264 21237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:54.431087 21237 log.cc:824] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:54.431345 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:54.427409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44821 (local address 127.0.0.1:33003)
0619 02:28:54.427811 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:54.427831 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:54.427871 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:28:54.428639 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:54.428662 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:54.429017 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:54.429347 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:54.430034 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:54.430039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:54.430053 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:54.430063 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:54.430128 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:54.430158 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:54.430288 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:54.430359 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 02:28:54.430522 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
I20260619 02:28:54.432911 21237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd: No bootstrap required, opened a new log
I20260619 02:28:54.433882 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:28:54.436722 21237 raft_consensus.cc:348] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de134e913b974fff813424526ba8f7fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33003 } }
I20260619 02:28:54.437016 21237 raft_consensus.cc:374] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:54.437079 21237 raft_consensus.cc:729] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de134e913b974fff813424526ba8f7fd, State: Initialized, Role: FOLLOWER
I20260619 02:28:54.437407 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:54.435868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58091 (local address 127.0.0.1:33003)
0619 02:28:54.435976 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:54.435979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:54.436145 (+ 166us) server_negotiation.cc:407] Connection header received
0619 02:28:54.436275 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:54.436278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:54.436313 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:54.436392 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:54.437044 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:54.437048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:54.437050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:54.437061 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:54.437087 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:54.437099 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:54.437171 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:54.437254 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 02:28:54.437290 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:28:54.437575 21237 consensus_queue.cc:260] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de134e913b974fff813424526ba8f7fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33003 } }
I20260619 02:28:54.437737 21237 raft_consensus.cc:388] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:54.437812 21237 raft_consensus.cc:482] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:54.437937 21237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:54.438887 21237 raft_consensus.cc:504] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de134e913b974fff813424526ba8f7fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33003 } }
I20260619 02:28:54.439201 21237 leader_election.cc:302] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de134e913b974fff813424526ba8f7fd; no voters:
I20260619 02:28:54.439482 21237 leader_election.cc:288] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:54.439605 21247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:54.439805 21247 raft_consensus.cc:686] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [term 1 LEADER]: Becoming Leader. State: Replica: de134e913b974fff813424526ba8f7fd, State: Running, Role: LEADER
I20260619 02:28:54.440120 21247 consensus_queue.cc:237] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de134e913b974fff813424526ba8f7fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33003 } }
I20260619 02:28:54.440629 21237 sys_catalog.cc:564] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:54.441682 21249 sys_catalog.cc:455] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader de134e913b974fff813424526ba8f7fd. Latest consensus state: current_term: 1 leader_uuid: "de134e913b974fff813424526ba8f7fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de134e913b974fff813424526ba8f7fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33003 } } }
I20260619 02:28:54.441812 21249 sys_catalog.cc:458] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:54.442092 21248 sys_catalog.cc:455] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de134e913b974fff813424526ba8f7fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de134e913b974fff813424526ba8f7fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33003 } } }
I20260619 02:28:54.442175 21256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:54.442198 21248 sys_catalog.cc:458] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:54.445107 21256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:54.450358 21256 catalog_manager.cc:1347] Generated new cluster ID: 461e4cc7a846476ea15ae4fb5fdf7860
I20260619 02:28:54.450444 21256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:54.461776 21256 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:54.462148 21256 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:54.471755 21256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de134e913b974fff813424526ba8f7fd: Generated new TSK 0
I20260619 02:28:54.472432 21256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:28:54.478431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 21169
2026-06-19T02:28:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:28:54 dist-test-slave-xkrt krb5kdc[21268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:28:54 dist-test-slave-xkrt krb5kdc[21268](info): set up 2 sockets
Jun 19 02:28:54 dist-test-slave-xkrt krb5kdc[21268](info): commencing operation
krb5kdc: starting...
W20260619 02:28:55.509805 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:28:55 dist-test-slave-xkrt krb5kdc[21268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:28:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:28:55.634671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:55.746511 21281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:55.746803 21281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:55.746891 21281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:55.750976 21281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:55.751053 21281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:55.751088 21281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:55.751144 21281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:55.751180 21281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:55.756536 21281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:55.757654 21281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:55.758711 21281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:55.767416 21286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:55.767341 21287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:55.767634 21289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:55.767467 21281 server_base.cc:1034] running on GCE node
I20260619 02:28:55.768251 21281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:55.769412 21281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:55.770608 21281 hybrid_clock.cc:648] HybridClock initialized: now 1781836135770597 us; error 30 us; skew 500 ppm
Jun 19 02:28:55 dist-test-slave-xkrt krb5kdc[21268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:55.777041 21281 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:28:55.778038 21281 webserver.cc:460] Webserver started at http://127.0.0.1:42129/ using document root <none> and password file <none>
I20260619 02:28:55.778623 21281 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:55.778716 21281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:55.778946 21281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:55.780990 21281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "ce5cb85dfd764b16aa5120b3907d220b"
format_stamp: "Formatted at 2026-06-19 02:28:55 on dist-test-slave-xkrt"
server_key: "15704ae3b91bd77f61d3204b3460002f"
server_key_iv: "ef27e8af0951b44f1ed51750b3d08b4a"
server_key_version: "encryptionkey@0"
I20260619 02:28:55.781610 21281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce5cb85dfd764b16aa5120b3907d220b"
format_stamp: "Formatted at 2026-06-19 02:28:55 on dist-test-slave-xkrt"
server_key: "15704ae3b91bd77f61d3204b3460002f"
server_key_iv: "ef27e8af0951b44f1ed51750b3d08b4a"
server_key_version: "encryptionkey@0"
I20260619 02:28:55.785179 21281 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:28:55.787592 21296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:55.788786 21281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:28:55.788930 21281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "ce5cb85dfd764b16aa5120b3907d220b"
format_stamp: "Formatted at 2026-06-19 02:28:55 on dist-test-slave-xkrt"
server_key: "15704ae3b91bd77f61d3204b3460002f"
server_key_iv: "ef27e8af0951b44f1ed51750b3d08b4a"
server_key_version: "encryptionkey@0"
I20260619 02:28:55.789068 21281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:55.801357 21281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:55.804250 21281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:55.804463 21281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:55.812031 21281 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45101
I20260619 02:28:55.812052 21348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45101 every 8 connection(s)
I20260619 02:28:55.812985 21281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:55.815719 21349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:55.820991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 21281
I20260619 02:28:55.821128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:55.821235 21349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b: Bootstrap starting.
I20260619 02:28:55.821410 392 external_mini_cluster.cc:1428] Setting key 3f5a60c99331fd554bf90a611e4a2a05
I20260619 02:28:55.823773 21349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:55.824525 21349 log.cc:824] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:55.826359 21349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b: No bootstrap required, opened a new log
I20260619 02:28:55.830088 21349 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5cb85dfd764b16aa5120b3907d220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45101 } }
Jun 19 02:28:55 dist-test-slave-xkrt krb5kdc[21268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:28:55.830374 21349 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:55.830428 21349 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5cb85dfd764b16aa5120b3907d220b, State: Initialized, Role: FOLLOWER
I20260619 02:28:55.830905 21349 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5cb85dfd764b16aa5120b3907d220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45101 } }
I20260619 02:28:55.831061 21349 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:55.831146 21349 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:55.831243 21349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:55.832372 21349 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5cb85dfd764b16aa5120b3907d220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45101 } }
I20260619 02:28:55.832707 21349 leader_election.cc:302] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce5cb85dfd764b16aa5120b3907d220b; no voters:
I20260619 02:28:55.833014 21349 leader_election.cc:288] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:55.833117 21354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:55.833391 21354 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [term 1 LEADER]: Becoming Leader. State: Replica: ce5cb85dfd764b16aa5120b3907d220b, State: Running, Role: LEADER
I20260619 02:28:55.833770 21354 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5cb85dfd764b16aa5120b3907d220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45101 } }
I20260619 02:28:55.834425 21349 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:55.835662 21356 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce5cb85dfd764b16aa5120b3907d220b. Latest consensus state: current_term: 1 leader_uuid: "ce5cb85dfd764b16aa5120b3907d220b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5cb85dfd764b16aa5120b3907d220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45101 } } }
I20260619 02:28:55.835722 21355 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce5cb85dfd764b16aa5120b3907d220b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5cb85dfd764b16aa5120b3907d220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45101 } } }
I20260619 02:28:55.835883 21356 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:55.835896 21355 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:55.837069 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:55.822820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:43483 (local address 127.0.0.1:45101)
0619 02:28:55.823245 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:55.823264 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:55.823299 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:28:55.824031 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:55.824050 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:55.824394 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:55.824728 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:55.825509 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:55.826597 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:55.827450 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:55.827949 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:55.828123 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:55.831025 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:55.831030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:55.831065 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:55.831102 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:55.833060 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:55.833713 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:55.833719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:55.833724 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:55.833853 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:55.834291 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:55.834294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:55.834296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:55.834796 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:55.835017 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:55.835683 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 02:28:55.835911 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":130,"threads_started":1}
I20260619 02:28:55.840548 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 02:28:55.841107 21369 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:28:55.841159 21369 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:28:55.841243 21365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:55.842129 21365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:55.847318 21365 catalog_manager.cc:1347] Generated new cluster ID: a5335e948b914c4da5491bf8cecd1102
I20260619 02:28:55.847381 21365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:55.852824 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:55.843071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:58199 (local address 127.0.0.1:45101)
0619 02:28:55.843169 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:55.843174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:55.843186 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:28:55.843223 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:55.843225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:55.843267 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:55.843351 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:55.844180 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:55.845002 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:55.846806 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:55.847180 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:55.847255 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:55.849168 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:55.849171 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:55.849174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:28:55.849197 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:55.850908 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:55.851456 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:55.851460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:55.851461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:55.851538 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:28:55.851874 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:28:55.851877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:28:55.851879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:28:55.852129 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:55.852276 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:55.852551 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 02:28:55.852655 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:28:55.860355 21365 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:55.861351 21365 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:55.867857 21365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce5cb85dfd764b16aa5120b3907d220b: Generated new TSK 0
I20260619 02:28:55.868571 21365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:28:55.874552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 21281
W20260619 02:28:55.879812 21266 connection.cc:535] client connection to 127.0.0.1:45101 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45101): Connection reset by peer (error 104)
2026-06-19T02:28:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:28:55 dist-test-slave-xkrt krb5kdc[21380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:28:55 dist-test-slave-xkrt krb5kdc[21380](info): set up 2 sockets
Jun 19 02:28:55 dist-test-slave-xkrt krb5kdc[21380](info): commencing operation
krb5kdc: starting...
W20260619 02:28:56.965842 392 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:28:57 dist-test-slave-xkrt krb5kdc[21380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:28:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:28:57.085703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32869
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:57.193743 21393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:57.194006 21393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:57.194047 21393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:57.198014 21393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:57.198071 21393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:57.198089 21393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:57.198105 21393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:57.198153 21393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:57.203318 21393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:57.204370 21393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:57.205379 21393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:57.213733 21399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:57.213778 21398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:57.213769 21401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:57.213821 21393 server_base.cc:1034] running on GCE node
I20260619 02:28:57.214484 21393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:57.215598 21393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:57.216787 21393 hybrid_clock.cc:648] HybridClock initialized: now 1781836137216776 us; error 26 us; skew 500 ppm
Jun 19 02:28:57 dist-test-slave-xkrt krb5kdc[21380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:57.219470 21393 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:28:57.220233 21393 webserver.cc:460] Webserver started at http://127.0.0.1:46477/ using document root <none> and password file <none>
I20260619 02:28:57.220788 21393 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:57.220849 21393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:57.221007 21393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:57.222950 21393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "472f3f91585a431692a10200e3cb0473"
format_stamp: "Formatted at 2026-06-19 02:28:57 on dist-test-slave-xkrt"
server_key: "3ecb16891fe12fef7ed10bc4ffef767f"
server_key_iv: "b902fae59b2c12448988d60ccd83cd6d"
server_key_version: "encryptionkey@0"
I20260619 02:28:57.223443 21393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "472f3f91585a431692a10200e3cb0473"
format_stamp: "Formatted at 2026-06-19 02:28:57 on dist-test-slave-xkrt"
server_key: "3ecb16891fe12fef7ed10bc4ffef767f"
server_key_iv: "b902fae59b2c12448988d60ccd83cd6d"
server_key_version: "encryptionkey@0"
I20260619 02:28:57.226907 21393 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:28:57.229079 21408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:57.230295 21393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:28:57.230422 21393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "472f3f91585a431692a10200e3cb0473"
format_stamp: "Formatted at 2026-06-19 02:28:57 on dist-test-slave-xkrt"
server_key: "3ecb16891fe12fef7ed10bc4ffef767f"
server_key_iv: "b902fae59b2c12448988d60ccd83cd6d"
server_key_version: "encryptionkey@0"
I20260619 02:28:57.230525 21393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:57.260056 21393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:57.263057 21393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:57.263260 21393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:57.269924 21393 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32869
I20260619 02:28:57.269958 21460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32869 every 8 connection(s)
I20260619 02:28:57.270859 21393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:57.271552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 21393
I20260619 02:28:57.271694 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:57.271979 392 external_mini_cluster.cc:1428] Setting key 14e13ca335cb05c554fb21eed5c55c55
I20260619 02:28:57.273901 21461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:57.279261 21461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473: Bootstrap starting.
I20260619 02:28:57.280773 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:57.273347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:43525 (local address 127.0.0.1:32869)
0619 02:28:57.273788 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:57.273803 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:57.273838 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:28:57.274560 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:57.274576 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:57.274946 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:57.275271 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:57.275648 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:57.276604 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:57.277486 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:57.278009 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:57.278167 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:57.279042 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:57.279047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:57.279058 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:57.279487 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:57.279537 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:57.279559 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:57.279694 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:57.279880 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 02:28:57.280022 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":148,"threads_started":1}
I20260619 02:28:57.281669 21461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:57.282517 21461 log.cc:824] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:57.283613 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:28:57.284617 21461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473: No bootstrap required, opened a new log
I20260619 02:28:57.288313 21461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472f3f91585a431692a10200e3cb0473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32869 } }
I20260619 02:28:57.288599 21461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:57.288662 21461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472f3f91585a431692a10200e3cb0473, State: Initialized, Role: FOLLOWER
I20260619 02:28:57.289134 21461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472f3f91585a431692a10200e3cb0473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32869 } }
I20260619 02:28:57.289315 21461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:57.289386 21461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:57.289482 21461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:57.290305 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:57.285815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:38189 (local address 127.0.0.1:32869)
0619 02:28:57.285945 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:57.285950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:57.286173 (+ 223us) server_negotiation.cc:407] Connection header received
0619 02:28:57.286275 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:57.286277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:57.286312 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:57.286379 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:57.286660 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:57.287485 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:57.288617 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:28:57.288917 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:28:57.288993 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:28:57.289853 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:57.289857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:57.289860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:57.289871 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:57.289901 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:57.289911 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:57.290004 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:57.290137 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 02:28:57.290171 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:28:57.290436 21461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472f3f91585a431692a10200e3cb0473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32869 } }
I20260619 02:28:57.290802 21461 leader_election.cc:302] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 472f3f91585a431692a10200e3cb0473; no voters:
I20260619 02:28:57.291136 21461 leader_election.cc:288] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:57.291249 21471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:57.291457 21471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [term 1 LEADER]: Becoming Leader. State: Replica: 472f3f91585a431692a10200e3cb0473, State: Running, Role: LEADER
I20260619 02:28:57.291772 21471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472f3f91585a431692a10200e3cb0473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32869 } }
I20260619 02:28:57.292344 21461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:57.293566 21473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 472f3f91585a431692a10200e3cb0473. Latest consensus state: current_term: 1 leader_uuid: "472f3f91585a431692a10200e3cb0473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472f3f91585a431692a10200e3cb0473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32869 } } }
I20260619 02:28:57.293684 21473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:57.293730 21472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "472f3f91585a431692a10200e3cb0473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472f3f91585a431692a10200e3cb0473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32869 } } }
I20260619 02:28:57.293826 21472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:57.294346 21480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:57.297329 21480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:57.302767 21480 catalog_manager.cc:1347] Generated new cluster ID: 82f71f49ae91411fbbfdd32c4b242aac
I20260619 02:28:57.302830 21480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:57.310083 21480 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:57.311165 21480 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:57.318516 21480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 472f3f91585a431692a10200e3cb0473: Generated new TSK 0
I20260619 02:28:57.319231 21480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:28:57.333324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 21393
W20260619 02:28:57.338743 21378 connection.cc:535] client connection to 127.0.0.1:32869 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32869): Connection reset by peer (error 104)
2026-06-19T02:28:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:28:57 dist-test-slave-xkrt krb5kdc[21492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:28:57 dist-test-slave-xkrt krb5kdc[21492](info): set up 2 sockets
Jun 19 02:28:57 dist-test-slave-xkrt krb5kdc[21492](info): commencing operation
krb5kdc: starting...
W20260619 02:28:58.365832 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:28:58 dist-test-slave-xkrt krb5kdc[21492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:28:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:28:58.482772 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:58.597725 21505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:58.598016 21505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:58.598098 21505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:58.602136 21505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:58.602218 21505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:58.602258 21505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:58.602312 21505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:58.602352 21505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:58.607589 21505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:58.608621 21505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:58.609670 21505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:28:58.618181 21511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:28:58.618304 21513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:58.618233 21505 server_base.cc:1034] running on GCE node
W20260619 02:28:58.618211 21510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:28:58.618847 21505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:28:58.619856 21505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:28:58.621037 21505 hybrid_clock.cc:648] HybridClock initialized: now 1781836138621020 us; error 33 us; skew 500 ppm
Jun 19 02:28:58 dist-test-slave-xkrt krb5kdc[21492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:28:58.623709 21505 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:28:58.624583 21505 webserver.cc:460] Webserver started at http://127.0.0.1:33179/ using document root <none> and password file <none>
I20260619 02:28:58.625082 21505 fs_manager.cc:359] Metadata directory not provided
I20260619 02:28:58.625137 21505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:28:58.625300 21505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:28:58.627283 21505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "2875fa58ce094130907b32357b471576"
format_stamp: "Formatted at 2026-06-19 02:28:58 on dist-test-slave-xkrt"
server_key: "bb6cc1c6a4e6e509d6f494a9f7ce2d34"
server_key_iv: "2ffbafb83baf30f06328a64b4a32224d"
server_key_version: "encryptionkey@0"
I20260619 02:28:58.627826 21505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "2875fa58ce094130907b32357b471576"
format_stamp: "Formatted at 2026-06-19 02:28:58 on dist-test-slave-xkrt"
server_key: "bb6cc1c6a4e6e509d6f494a9f7ce2d34"
server_key_iv: "2ffbafb83baf30f06328a64b4a32224d"
server_key_version: "encryptionkey@0"
I20260619 02:28:58.631435 21505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:28:58.633865 21520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:28:58.635089 21505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:28:58.635224 21505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "2875fa58ce094130907b32357b471576"
format_stamp: "Formatted at 2026-06-19 02:28:58 on dist-test-slave-xkrt"
server_key: "bb6cc1c6a4e6e509d6f494a9f7ce2d34"
server_key_iv: "2ffbafb83baf30f06328a64b4a32224d"
server_key_version: "encryptionkey@0"
I20260619 02:28:58.635330 21505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:28:58.639787 21505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:28:58.642659 21505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:28:58.642844 21505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:28:58.649714 21505 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34987
I20260619 02:28:58.649739 21572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34987 every 8 connection(s)
I20260619 02:28:58.650591 21505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:28:58.653179 21573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:28:58.658203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 21505
I20260619 02:28:58.658342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:28:58.658582 21573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576: Bootstrap starting.
I20260619 02:28:58.658615 392 external_mini_cluster.cc:1428] Setting key 9146ebec8ecccf23fcdebe83dde4071e
I20260619 02:28:58.660987 21573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576: Neither blocks nor log segments found. Creating new log.
I20260619 02:28:58.661814 21573 log.cc:824] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576: Log is configured to *not* fsync() on all Append() calls
I20260619 02:28:58.663509 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:58.659861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:32857 (local address 127.0.0.1:34987)
0619 02:28:58.660262 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:58.660275 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:58.660308 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:28:58.660920 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:58.660936 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:58.661244 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:58.661596 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:58.662359 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:58.662364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:58.662375 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:58.662385 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:58.662435 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:58.662454 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:58.662567 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:58.662683 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 02:28:58.662870 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260619 02:28:58.663802 21573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576: No bootstrap required, opened a new log
I20260619 02:28:58.666365 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:28:58.667937 21573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2875fa58ce094130907b32357b471576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34987 } }
I20260619 02:28:58.668166 21573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:28:58.668226 21573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2875fa58ce094130907b32357b471576, State: Initialized, Role: FOLLOWER
I20260619 02:28:58.668650 21573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2875fa58ce094130907b32357b471576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34987 } }
I20260619 02:28:58.668803 21573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:28:58.668895 21573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:28:58.668998 21573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:28:58.670006 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:28:58.668307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:38387 (local address 127.0.0.1:34987)
0619 02:28:58.668424 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:28:58.668428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:28:58.668642 (+ 214us) server_negotiation.cc:407] Connection header received
0619 02:28:58.668778 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:28:58.668781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:28:58.668812 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:28:58.668885 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:28:58.669627 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:28:58.669632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:28:58.669639 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:28:58.669649 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:28:58.669672 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:28:58.669682 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:28:58.669748 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:28:58.669864 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 02:28:58.669892 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:28:58.670203 21573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2875fa58ce094130907b32357b471576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34987 } }
I20260619 02:28:58.670553 21573 leader_election.cc:302] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2875fa58ce094130907b32357b471576; no voters:
I20260619 02:28:58.670892 21573 leader_election.cc:288] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:28:58.670974 21583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:28:58.671140 21583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [term 1 LEADER]: Becoming Leader. State: Replica: 2875fa58ce094130907b32357b471576, State: Running, Role: LEADER
I20260619 02:28:58.671453 21583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2875fa58ce094130907b32357b471576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34987 } }
I20260619 02:28:58.672083 21573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:28:58.673094 21585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2875fa58ce094130907b32357b471576. Latest consensus state: current_term: 1 leader_uuid: "2875fa58ce094130907b32357b471576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2875fa58ce094130907b32357b471576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34987 } } }
I20260619 02:28:58.673215 21585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:58.673348 21584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2875fa58ce094130907b32357b471576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2875fa58ce094130907b32357b471576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34987 } } }
I20260619 02:28:58.673532 21584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576 [sys.catalog]: This master's current role is: LEADER
I20260619 02:28:58.673777 21592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:28:58.676518 21592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:28:58.681800 21592 catalog_manager.cc:1347] Generated new cluster ID: f93f2d0f0b244c6c85cc5624fb5add24
I20260619 02:28:58.681875 21592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:28:58.699641 21592 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:28:58.699961 21592 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:28:58.713152 21592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2875fa58ce094130907b32357b471576: Generated new TSK 0
I20260619 02:28:58.713912 21592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:28:58.723188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 21505
2026-06-19T02:28:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:28:58 dist-test-slave-xkrt krb5kdc[21604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:28:58 dist-test-slave-xkrt krb5kdc[21604](info): set up 2 sockets
Jun 19 02:28:58 dist-test-slave-xkrt krb5kdc[21604](info): commencing operation
krb5kdc: starting...
W20260619 02:28:59.765699 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:28:59 dist-test-slave-xkrt krb5kdc[21604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:28:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:28:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:28:59.879988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:28:59.984709 21617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:28:59.984939 21617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:28:59.984978 21617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:28:59.988929 21617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:28:59.988981 21617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:28:59.989001 21617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:28:59.989017 21617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:28:59.989032 21617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:28:59.994058 21617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37671
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:28:59.995012 21617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:28:59.995931 21617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:00.004225 21625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:00.004225 21622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:00.004282 21623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:00.004623 21617 server_base.cc:1034] running on GCE node
I20260619 02:29:00.005143 21617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:00.006274 21617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:00.007442 21617 hybrid_clock.cc:648] HybridClock initialized: now 1781836140007416 us; error 41 us; skew 500 ppm
Jun 19 02:29:00 dist-test-slave-xkrt krb5kdc[21604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:00.010118 21617 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:29:00.010979 21617 webserver.cc:460] Webserver started at http://127.0.0.1:39143/ using document root <none> and password file <none>
I20260619 02:29:00.011515 21617 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:00.011576 21617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:00.011817 21617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:00.013827 21617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "7004717f3750478b94895916c7b327ef"
format_stamp: "Formatted at 2026-06-19 02:29:00 on dist-test-slave-xkrt"
server_key: "dfcc161bea47d4b6d11b16e31387e35b"
server_key_iv: "9b44f89c61946759cbdfae702567cb3b"
server_key_version: "encryptionkey@0"
I20260619 02:29:00.014374 21617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "7004717f3750478b94895916c7b327ef"
format_stamp: "Formatted at 2026-06-19 02:29:00 on dist-test-slave-xkrt"
server_key: "dfcc161bea47d4b6d11b16e31387e35b"
server_key_iv: "9b44f89c61946759cbdfae702567cb3b"
server_key_version: "encryptionkey@0"
I20260619 02:29:00.017947 21617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:29:00.020165 21632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:00.021351 21617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:29:00.021485 21617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "7004717f3750478b94895916c7b327ef"
format_stamp: "Formatted at 2026-06-19 02:29:00 on dist-test-slave-xkrt"
server_key: "dfcc161bea47d4b6d11b16e31387e35b"
server_key_iv: "9b44f89c61946759cbdfae702567cb3b"
server_key_version: "encryptionkey@0"
I20260619 02:29:00.021641 21617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:00.043465 21617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:00.046187 21617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:00.046377 21617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:00.053189 21617 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37671
I20260619 02:29:00.053210 21684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37671 every 8 connection(s)
I20260619 02:29:00.054095 21617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:29:00.055406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 21617
I20260619 02:29:00.055562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:29:00.055874 392 external_mini_cluster.cc:1428] Setting key f5e63c31c06dfe9cfb313cc939adc971
I20260619 02:29:00.057237 21685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:00.063565 21685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef: Bootstrap starting.
Jun 19 02:29:00 dist-test-slave-xkrt krb5kdc[21604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:29:00.065830 21685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:00.066524 21685 log.cc:824] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:00.068275 21685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef: No bootstrap required, opened a new log
I20260619 02:29:00.069633 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:00.057121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:56493 (local address 127.0.0.1:37671)
0619 02:29:00.057489 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:00.057513 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:00.057562 (+ 49us) server_negotiation.cc:407] Connection header received
0619 02:29:00.058204 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:00.058219 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:00.058534 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:00.058797 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:00.059637 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:00.060586 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:00.061750 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:00.062189 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:00.062298 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:00.064998 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:00.065003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:00.065014 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:00.065038 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:00.066783 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:00.067302 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:00.067307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:00.067319 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:00.067425 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:00.067750 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:00.067753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:00.067755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:00.068115 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:00.068330 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:00.068676 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 02:29:00.068920 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":127,"threads_started":1}
I20260619 02:29:00.072480 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:29:00.072297 21685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7004717f3750478b94895916c7b327ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } }
I20260619 02:29:00.072554 21685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:29:00.072608 21685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7004717f3750478b94895916c7b327ef, State: Initialized, Role: FOLLOWER
I20260619 02:29:00.073125 21685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7004717f3750478b94895916c7b327ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } }
I20260619 02:29:00.073282 21685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:29:00.073362 21685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:29:00.073460 21685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:29:00.074649 21685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7004717f3750478b94895916c7b327ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } }
I20260619 02:29:00.074998 21685 leader_election.cc:302] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7004717f3750478b94895916c7b327ef; no voters:
I20260619 02:29:00.075316 21685 leader_election.cc:288] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:29:00.075426 21695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:29:00.075680 21695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [term 1 LEADER]: Becoming Leader. State: Replica: 7004717f3750478b94895916c7b327ef, State: Running, Role: LEADER
I20260619 02:29:00.075995 21695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7004717f3750478b94895916c7b327ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } }
I20260619 02:29:00.076692 21685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:00.077422 21696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7004717f3750478b94895916c7b327ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7004717f3750478b94895916c7b327ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } } }
I20260619 02:29:00.077561 21696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:00.077769 21696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7004717f3750478b94895916c7b327ef. Latest consensus state: current_term: 1 leader_uuid: "7004717f3750478b94895916c7b327ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7004717f3750478b94895916c7b327ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } } }
I20260619 02:29:00.077840 21696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:00.078132 21704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:29:00.081405 21704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:29:00.084246 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:00.074734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:60973 (local address 127.0.0.1:37671)
0619 02:29:00.074898 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:00.074902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:00.075098 (+ 196us) server_negotiation.cc:407] Connection header received
0619 02:29:00.075229 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:00.075233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:00.075275 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:00.075374 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:00.076165 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:00.076916 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:00.078183 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:00.078572 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:00.078660 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:00.080493 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:00.080499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:00.080504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:00.080530 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:00.082247 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:00.082828 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:00.082832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:00.082833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:00.082924 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:00.083294 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:00.083297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:00.083299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:00.083500 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:00.083645 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:00.083969 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 02:29:00.084087 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260619 02:29:00.087888 21704 catalog_manager.cc:1347] Generated new cluster ID: e5b6530e11e04e3d9a25bdb81b43442b
I20260619 02:29:00.087952 21704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:29:00.104514 21704 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:29:00.105878 21704 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:29:00.112352 21704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7004717f3750478b94895916c7b327ef: Generated new TSK 0
I20260619 02:29:00.113129 21704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:29:00.120945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 21617
W20260619 02:29:00.126026 21602 connection.cc:535] client connection to 127.0.0.1:37671 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37671): Connection reset by peer (error 104)
2026-06-19T02:29:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:29:00 dist-test-slave-xkrt krb5kdc[21716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:29:00 dist-test-slave-xkrt krb5kdc[21716](info): set up 2 sockets
Jun 19 02:29:00 dist-test-slave-xkrt krb5kdc[21716](info): commencing operation
krb5kdc: starting...
W20260619 02:29:01.161708 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:29:01 dist-test-slave-xkrt krb5kdc[21716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:29:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:29:01.282859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:29:01.394121 21729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:29:01.394381 21729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:29:01.394451 21729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:29:01.398519 21729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:29:01.398571 21729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:29:01.398590 21729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:29:01.398607 21729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:29:01.398651 21729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:29:01.403800 21729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:29:01.404757 21729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:29:01.405915 21729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:01.414045 21737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:01.414114 21734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:01.414124 21735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:01.414307 21729 server_base.cc:1034] running on GCE node
I20260619 02:29:01.414935 21729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:01.415992 21729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:01.417183 21729 hybrid_clock.cc:648] HybridClock initialized: now 1781836141417160 us; error 43 us; skew 500 ppm
Jun 19 02:29:01 dist-test-slave-xkrt krb5kdc[21716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:01.423005 21729 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:29:01.423868 21729 webserver.cc:460] Webserver started at http://127.0.0.1:36261/ using document root <none> and password file <none>
I20260619 02:29:01.424428 21729 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:01.424511 21729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:01.424721 21729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:01.426704 21729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "adf207d111524ea68d4985f05a4633b1"
format_stamp: "Formatted at 2026-06-19 02:29:01 on dist-test-slave-xkrt"
server_key: "67a5dbd5871a48ac28990005842025a7"
server_key_iv: "c2fbd3a6d71e03e184b19a42c1703213"
server_key_version: "encryptionkey@0"
I20260619 02:29:01.427234 21729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "adf207d111524ea68d4985f05a4633b1"
format_stamp: "Formatted at 2026-06-19 02:29:01 on dist-test-slave-xkrt"
server_key: "67a5dbd5871a48ac28990005842025a7"
server_key_iv: "c2fbd3a6d71e03e184b19a42c1703213"
server_key_version: "encryptionkey@0"
I20260619 02:29:01.430764 21729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:29:01.433058 21744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:01.434221 21729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:29:01.434358 21729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "adf207d111524ea68d4985f05a4633b1"
format_stamp: "Formatted at 2026-06-19 02:29:01 on dist-test-slave-xkrt"
server_key: "67a5dbd5871a48ac28990005842025a7"
server_key_iv: "c2fbd3a6d71e03e184b19a42c1703213"
server_key_version: "encryptionkey@0"
I20260619 02:29:01.434463 21729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:01.448601 21729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:01.451658 21729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:01.451858 21729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:01.458570 21729 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42705
I20260619 02:29:01.458583 21796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42705 every 8 connection(s)
I20260619 02:29:01.459447 21729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:29:01.462038 21797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:01.467193 21797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1: Bootstrap starting.
I20260619 02:29:01.468441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 21729
I20260619 02:29:01.468591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:29:01.468873 392 external_mini_cluster.cc:1428] Setting key 4d8ff1ffad30628602b32a2fae0a0f8d
I20260619 02:29:01.469662 21797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:01.470496 21797 log.cc:824] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:01.472484 21797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1: No bootstrap required, opened a new log
I20260619 02:29:01.476609 21797 raft_consensus.cc:348] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf207d111524ea68d4985f05a4633b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } }
I20260619 02:29:01.476897 21797 raft_consensus.cc:374] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:29:01.476977 21797 raft_consensus.cc:729] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf207d111524ea68d4985f05a4633b1, State: Initialized, Role: FOLLOWER
I20260619 02:29:01.477416 21797 consensus_queue.cc:260] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf207d111524ea68d4985f05a4633b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } }
I20260619 02:29:01.477583 21797 raft_consensus.cc:388] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:29:01.477672 21797 raft_consensus.cc:482] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:29:01.477768 21797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:29:01.477841 21800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:01.469919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:46785 (local address 127.0.0.1:42705)
0619 02:29:01.470294 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:01.470309 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:01.470341 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:29:01.470991 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:01.471009 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:01.471333 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:01.471595 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:01.471946 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:01.473017 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:01.474259 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:01.474759 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:01.474896 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:01.475836 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:01.475841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:01.475854 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:29:01.476241 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:01.476306 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:29:01.476329 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:01.476432 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:01.476640 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 02:29:01.476803 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":115,"threads_started":1}
I20260619 02:29:01.478914 21797 raft_consensus.cc:504] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf207d111524ea68d4985f05a4633b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } }
I20260619 02:29:01.479259 21797 leader_election.cc:302] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adf207d111524ea68d4985f05a4633b1; no voters:
I20260619 02:29:01.479571 21797 leader_election.cc:288] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:29:01.479702 21802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:29:01.479969 21802 raft_consensus.cc:686] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [term 1 LEADER]: Becoming Leader. State: Replica: adf207d111524ea68d4985f05a4633b1, State: Running, Role: LEADER
I20260619 02:29:01.480335 21802 consensus_queue.cc:237] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf207d111524ea68d4985f05a4633b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } }
I20260619 02:29:01.480813 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:29:01.480885 21797 sys_catalog.cc:564] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:01.482170 21804 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adf207d111524ea68d4985f05a4633b1. Latest consensus state: current_term: 1 leader_uuid: "adf207d111524ea68d4985f05a4633b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf207d111524ea68d4985f05a4633b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } } }
I20260619 02:29:01.482282 21804 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:01.482635 21803 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adf207d111524ea68d4985f05a4633b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf207d111524ea68d4985f05a4633b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } } }
I20260619 02:29:01.483364 21803 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1 [sys.catalog]: This master's current role is: LEADER
W20260619 02:29:01.485754 21821 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:29:01.485877 21821 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:29:01.485946 21815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:29:01.486727 21815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:29:01.488694 21800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:01.483939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:55677 (local address 127.0.0.1:42705)
0619 02:29:01.484054 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:01.484062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:01.484304 (+ 242us) server_negotiation.cc:407] Connection header received
0619 02:29:01.484437 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:01.484440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:01.484488 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:01.484571 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:01.484852 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:01.485668 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:01.486788 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:01.487133 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:01.487245 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:01.488185 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:01.488189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:01.488192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:29:01.488205 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:01.488231 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:29:01.488245 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:01.488347 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:01.488525 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 02:29:01.488561 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:29:01.492246 21815 catalog_manager.cc:1347] Generated new cluster ID: 608acd229d5b4618a39a81a5bc1395c6
I20260619 02:29:01.492305 21815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:29:01.514271 21815 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:29:01.515177 21815 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:29:01.529582 21815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adf207d111524ea68d4985f05a4633b1: Generated new TSK 0
I20260619 02:29:01.530237 21815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:29:01.542887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 21729
W20260619 02:29:01.548449 21714 connection.cc:535] client connection to 127.0.0.1:42705 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42705): Connection reset by peer (error 104)
2026-06-19T02:29:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:29:01 dist-test-slave-xkrt krb5kdc[21828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:29:01 dist-test-slave-xkrt krb5kdc[21828](info): set up 2 sockets
Jun 19 02:29:01 dist-test-slave-xkrt krb5kdc[21828](info): commencing operation
krb5kdc: starting...
W20260619 02:29:02.593766 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:29:02 dist-test-slave-xkrt krb5kdc[21828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:29:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:29:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:29:02.708608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:29:02.815136 21841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:29:02.815449 21841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:29:02.815536 21841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:29:02.819445 21841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:29:02.819531 21841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:29:02.819571 21841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:29:02.819629 21841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:29:02.819667 21841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:29:02.824692 21841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:29:02.825716 21841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:29:02.826691 21841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:02.835090 21846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:02.835054 21849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:02.835100 21841 server_base.cc:1034] running on GCE node
W20260619 02:29:02.835210 21847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:02.835953 21841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:02.836933 21841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:02.838099 21841 hybrid_clock.cc:648] HybridClock initialized: now 1781836142838081 us; error 29 us; skew 500 ppm
Jun 19 02:29:02 dist-test-slave-xkrt krb5kdc[21828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:02.840775 21841 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:29:02.841625 21841 webserver.cc:460] Webserver started at http://127.0.0.1:41953/ using document root <none> and password file <none>
I20260619 02:29:02.842133 21841 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:02.842191 21841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:02.842343 21841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:02.844278 21841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "6d4f073c8cad448dbdee808cf5d9ec7c"
format_stamp: "Formatted at 2026-06-19 02:29:02 on dist-test-slave-xkrt"
server_key: "e176f5d6de48d976d0d891080b6f0fcb"
server_key_iv: "70d896965a134d0fd0b111ffd6d56cca"
server_key_version: "encryptionkey@0"
I20260619 02:29:02.844774 21841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "6d4f073c8cad448dbdee808cf5d9ec7c"
format_stamp: "Formatted at 2026-06-19 02:29:02 on dist-test-slave-xkrt"
server_key: "e176f5d6de48d976d0d891080b6f0fcb"
server_key_iv: "70d896965a134d0fd0b111ffd6d56cca"
server_key_version: "encryptionkey@0"
I20260619 02:29:02.848379 21841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:29:02.850715 21856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:02.851871 21841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:29:02.852008 21841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "6d4f073c8cad448dbdee808cf5d9ec7c"
format_stamp: "Formatted at 2026-06-19 02:29:02 on dist-test-slave-xkrt"
server_key: "e176f5d6de48d976d0d891080b6f0fcb"
server_key_iv: "70d896965a134d0fd0b111ffd6d56cca"
server_key_version: "encryptionkey@0"
I20260619 02:29:02.852119 21841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:02.856542 21841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:02.859234 21841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:02.859416 21841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:02.866031 21841 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37381
I20260619 02:29:02.866060 21908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37381 every 8 connection(s)
I20260619 02:29:02.866879 21841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:29:02.869321 21909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:02.873746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 21841
I20260619 02:29:02.873886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:29:02.874152 392 external_mini_cluster.cc:1428] Setting key cb5cdffcf462f35cfaf2bb22214525e1
I20260619 02:29:02.874696 21909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c: Bootstrap starting.
I20260619 02:29:02.877118 21909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:02.877979 21909 log.cc:824] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:02.879223 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:02.875457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:49915 (local address 127.0.0.1:37381)
0619 02:29:02.875872 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:02.875889 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:02.875923 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:29:02.876637 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:02.876655 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:02.876949 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:02.877287 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:02.878031 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:02.878036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:02.878057 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:29:02.878066 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:02.878141 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:29:02.878166 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:02.878291 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:02.878408 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 02:29:02.878561 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":123,"threads_started":1}
I20260619 02:29:02.879920 21909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c: No bootstrap required, opened a new log
I20260619 02:29:02.881913 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:29:02.883823 21909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4f073c8cad448dbdee808cf5d9ec7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } }
I20260619 02:29:02.884114 21909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:29:02.884172 21909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4f073c8cad448dbdee808cf5d9ec7c, State: Initialized, Role: FOLLOWER
I20260619 02:29:02.884593 21909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4f073c8cad448dbdee808cf5d9ec7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } }
I20260619 02:29:02.884732 21909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:29:02.884804 21909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:29:02.884909 21909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:29:02.885637 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:02.883859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:36121 (local address 127.0.0.1:37381)
0619 02:29:02.883978 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:02.883981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:02.884204 (+ 223us) server_negotiation.cc:407] Connection header received
0619 02:29:02.884422 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:02.884425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:02.884456 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:02.884547 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:02.885240 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:02.885243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:02.885246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:29:02.885256 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:02.885281 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:29:02.885292 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:02.885393 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:02.885468 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 02:29:02.885506 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:29:02.885913 21909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4f073c8cad448dbdee808cf5d9ec7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } }
I20260619 02:29:02.886232 21909 leader_election.cc:302] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d4f073c8cad448dbdee808cf5d9ec7c; no voters:
I20260619 02:29:02.886546 21909 leader_election.cc:288] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:29:02.886653 21919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:29:02.886863 21919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [term 1 LEADER]: Becoming Leader. State: Replica: 6d4f073c8cad448dbdee808cf5d9ec7c, State: Running, Role: LEADER
I20260619 02:29:02.887188 21919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4f073c8cad448dbdee808cf5d9ec7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } }
I20260619 02:29:02.887791 21909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:02.888751 21921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d4f073c8cad448dbdee808cf5d9ec7c. Latest consensus state: current_term: 1 leader_uuid: "6d4f073c8cad448dbdee808cf5d9ec7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4f073c8cad448dbdee808cf5d9ec7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } } }
I20260619 02:29:02.888873 21921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:02.889168 21920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d4f073c8cad448dbdee808cf5d9ec7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4f073c8cad448dbdee808cf5d9ec7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } } }
I20260619 02:29:02.889274 21920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:02.889611 21928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:29:02.892316 21928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:29:02.897784 21928 catalog_manager.cc:1347] Generated new cluster ID: 01b01d5bd164495b8563ef81130c257b
I20260619 02:29:02.897845 21928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:29:02.921743 21928 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:29:02.922075 21928 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:29:02.935688 21928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d4f073c8cad448dbdee808cf5d9ec7c: Generated new TSK 0
I20260619 02:29:02.936470 21928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:29:02.941421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 21841
2026-06-19T02:29:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:29:02 dist-test-slave-xkrt krb5kdc[21940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:29:02 dist-test-slave-xkrt krb5kdc[21940](info): set up 2 sockets
Jun 19 02:29:02 dist-test-slave-xkrt krb5kdc[21940](info): commencing operation
krb5kdc: starting...
W20260619 02:29:03.973745 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:29:04 dist-test-slave-xkrt krb5kdc[21940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:29:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:29:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:29:04.092545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:29:04.198524 21953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:29:04.198815 21953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:29:04.198905 21953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:29:04.202858 21953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:29:04.202946 21953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:29:04.202986 21953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:29:04.203044 21953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:29:04.203085 21953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:29:04.208158 21953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:29:04.209184 21953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:29:04.210191 21953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:04.218377 21958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:04.218395 21961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:04.218516 21953 server_base.cc:1034] running on GCE node
W20260619 02:29:04.218396 21959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:04.219151 21953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:04.220265 21953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:04.221451 21953 hybrid_clock.cc:648] HybridClock initialized: now 1781836144221425 us; error 40 us; skew 500 ppm
Jun 19 02:29:04 dist-test-slave-xkrt krb5kdc[21940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:04.224084 21953 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:29:04.224968 21953 webserver.cc:460] Webserver started at http://127.0.98.62:37273/ using document root <none> and password file <none>
I20260619 02:29:04.225523 21953 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:04.225669 21953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:04.225911 21953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:04.227932 21953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "b52a0439480d411e8eed312f067d8ebc"
format_stamp: "Formatted at 2026-06-19 02:29:04 on dist-test-slave-xkrt"
server_key: "92817e7d8d22f946424b0f1d24f1b036"
server_key_iv: "af9940bc555f7d5030fa28c1b1d9bd6e"
server_key_version: "encryptionkey@0"
I20260619 02:29:04.228498 21953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "b52a0439480d411e8eed312f067d8ebc"
format_stamp: "Formatted at 2026-06-19 02:29:04 on dist-test-slave-xkrt"
server_key: "92817e7d8d22f946424b0f1d24f1b036"
server_key_iv: "af9940bc555f7d5030fa28c1b1d9bd6e"
server_key_version: "encryptionkey@0"
I20260619 02:29:04.232208 21953 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260619 02:29:04.234635 21968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:04.235865 21953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:29:04.236011 21953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "b52a0439480d411e8eed312f067d8ebc"
format_stamp: "Formatted at 2026-06-19 02:29:04 on dist-test-slave-xkrt"
server_key: "92817e7d8d22f946424b0f1d24f1b036"
server_key_iv: "af9940bc555f7d5030fa28c1b1d9bd6e"
server_key_version: "encryptionkey@0"
I20260619 02:29:04.236129 21953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:04.257697 21953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:04.260649 21953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:04.260871 21953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:04.267885 21953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37889
I20260619 02:29:04.267953 22020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37889 every 8 connection(s)
I20260619 02:29:04.268818 21953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:29:04.271467 22021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:04.276686 22021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc: Bootstrap starting.
I20260619 02:29:04.278100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 21953
I20260619 02:29:04.278213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:29:04.278501 392 external_mini_cluster.cc:1428] Setting key b8ab5457a708d36c686125370edb9a1c
I20260619 02:29:04.279179 22021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:04.280043 22021 log.cc:824] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:04.281853 22021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc: No bootstrap required, opened a new log
I20260619 02:29:04.285115 22021 raft_consensus.cc:348] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b52a0439480d411e8eed312f067d8ebc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260619 02:29:04.285396 22021 raft_consensus.cc:374] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:29:04.285481 22021 raft_consensus.cc:729] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52a0439480d411e8eed312f067d8ebc, State: Initialized, Role: FOLLOWER
I20260619 02:29:04.285925 22021 consensus_queue.cc:260] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b52a0439480d411e8eed312f067d8ebc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260619 02:29:04.286082 22021 raft_consensus.cc:388] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:29:04.286170 22021 raft_consensus.cc:482] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:29:04.286266 22021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [term 0 FOLLOWER]: Advancing to term 1
Jun 19 02:29:04 dist-test-slave-xkrt krb5kdc[21940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:29:04.287176 22021 raft_consensus.cc:504] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b52a0439480d411e8eed312f067d8ebc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260619 02:29:04.287503 22021 leader_election.cc:302] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b52a0439480d411e8eed312f067d8ebc; no voters:
I20260619 02:29:04.287822 22021 leader_election.cc:288] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:29:04.287890 22026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:29:04.288089 22026 raft_consensus.cc:686] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [term 1 LEADER]: Becoming Leader. State: Replica: b52a0439480d411e8eed312f067d8ebc, State: Running, Role: LEADER
I20260619 02:29:04.288429 22026 consensus_queue.cc:237] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b52a0439480d411e8eed312f067d8ebc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260619 02:29:04.289057 22021 sys_catalog.cc:564] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:04.290269 22027 sys_catalog.cc:455] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b52a0439480d411e8eed312f067d8ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b52a0439480d411e8eed312f067d8ebc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } }
I20260619 02:29:04.290274 22028 sys_catalog.cc:455] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b52a0439480d411e8eed312f067d8ebc. Latest consensus state: current_term: 1 leader_uuid: "b52a0439480d411e8eed312f067d8ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b52a0439480d411e8eed312f067d8ebc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } }
I20260619 02:29:04.290483 22027 sys_catalog.cc:458] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:04.290490 22028 sys_catalog.cc:458] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:04.290949 22036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:29:04.293128 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:04.279503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39517 (local address 127.0.98.62:37889)
0619 02:29:04.279926 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:04.279941 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:04.279975 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:29:04.280629 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:04.280646 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:04.280967 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:04.281295 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:04.282148 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:04.283077 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:04.283959 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:04.284430 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:04.284595 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:04.287376 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:04.287382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:04.287395 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:04.287420 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:04.289503 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:04.290187 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:04.290195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:04.290201 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:04.290358 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:04.290772 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:04.290775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:04.290777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:04.291272 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:04.291515 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:04.292002 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 02:29:04.292236 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
I20260619 02:29:04.294951 22036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:29:04.296227 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:29:04.300642 22036 catalog_manager.cc:1347] Generated new cluster ID: ce09f00728534e918e13d82c4440152a
I20260619 02:29:04.300704 22036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:29:04.307094 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:04.299484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51549 (local address 127.0.98.62:37889)
0619 02:29:04.299585 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:04.299589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:04.299858 (+ 269us) server_negotiation.cc:407] Connection header received
0619 02:29:04.299962 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:04.299966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:04.300009 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:04.300106 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:04.300904 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:04.301420 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:04.302270 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:04.302507 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:04.302618 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:04.303947 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:04.303951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:04.303954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:04.303984 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:04.305347 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:04.305844 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:04.305848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:04.305852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:04.305925 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:04.306246 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:04.306249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:04.306251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:04.306413 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:04.306577 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:04.306833 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 02:29:04.306919 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:29:04.318017 22036 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:29:04.318897 22036 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:29:04.325146 22036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b52a0439480d411e8eed312f067d8ebc: Generated new TSK 0
I20260619 02:29:04.325843 22036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:29:04.333456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 21953
W20260619 02:29:04.338775 21938 connection.cc:535] client connection to 127.0.98.62:37889 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37889): Connection reset by peer (error 104)
2026-06-19T02:29:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:29:04 dist-test-slave-xkrt krb5kdc[22052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:29:04 dist-test-slave-xkrt krb5kdc[22052](info): set up 2 sockets
Jun 19 02:29:04 dist-test-slave-xkrt krb5kdc[22052](info): commencing operation
krb5kdc: starting...
W20260619 02:29:05.377727 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:29:05 dist-test-slave-xkrt krb5kdc[22052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:29:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:29:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:29:05.496160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34855
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:29:05.610239 22065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:29:05.610563 22065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:29:05.610673 22065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:29:05.615566 22065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:29:05.615617 22065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:29:05.615636 22065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:29:05.615653 22065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:29:05.615713 22065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:29:05.620762 22065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:29:05.621984 22065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:29:05.623087 22065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:05.631266 22073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:05.631312 22070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:05.631263 22071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:05.631440 22065 server_base.cc:1034] running on GCE node
I20260619 02:29:05.632117 22065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:05.633147 22065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:05.634341 22065 hybrid_clock.cc:648] HybridClock initialized: now 1781836145634323 us; error 29 us; skew 500 ppm
Jun 19 02:29:05 dist-test-slave-xkrt krb5kdc[22052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:05.637042 22065 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:29:05.637997 22065 webserver.cc:460] Webserver started at http://127.0.98.62:43549/ using document root <none> and password file <none>
I20260619 02:29:05.638513 22065 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:05.638602 22065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:05.638845 22065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:05.640753 22065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "f0df6294193142d48dcc86ff652627ff"
format_stamp: "Formatted at 2026-06-19 02:29:05 on dist-test-slave-xkrt"
server_key: "d763103af304ab4b74a1ba0f685258e9"
server_key_iv: "36e72b363e1033d9c2239d921953aad1"
server_key_version: "encryptionkey@0"
I20260619 02:29:05.641300 22065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0df6294193142d48dcc86ff652627ff"
format_stamp: "Formatted at 2026-06-19 02:29:05 on dist-test-slave-xkrt"
server_key: "d763103af304ab4b74a1ba0f685258e9"
server_key_iv: "36e72b363e1033d9c2239d921953aad1"
server_key_version: "encryptionkey@0"
I20260619 02:29:05.644778 22065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:29:05.647181 22080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:05.648311 22065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:29:05.648450 22065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "f0df6294193142d48dcc86ff652627ff"
format_stamp: "Formatted at 2026-06-19 02:29:05 on dist-test-slave-xkrt"
server_key: "d763103af304ab4b74a1ba0f685258e9"
server_key_iv: "36e72b363e1033d9c2239d921953aad1"
server_key_version: "encryptionkey@0"
I20260619 02:29:05.648557 22065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:05.662880 22065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:05.665983 22065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:05.666178 22065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:05.673049 22065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34855
I20260619 02:29:05.673079 22132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34855 every 8 connection(s)
I20260619 02:29:05.673955 22065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:29:05.676592 22133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:05.681915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 22065
I20260619 02:29:05.681967 22133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff: Bootstrap starting.
I20260619 02:29:05.682060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:29:05.682343 392 external_mini_cluster.cc:1428] Setting key fd493a10d92e81615e8b9025427872c3
I20260619 02:29:05.684518 22133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:05.685268 22133 log.cc:824] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:05.687238 22133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff: No bootstrap required, opened a new log
I20260619 02:29:05.690936 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:05.683542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46383 (local address 127.0.98.62:34855)
0619 02:29:05.683949 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:05.683965 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:05.683999 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:29:05.684594 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:05.684610 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:05.684969 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:05.685227 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:05.685656 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:05.686594 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:05.687812 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:05.688264 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:05.688378 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:05.689272 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:05.689276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:05.689288 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:29:05.689712 (+ 424us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:05.689761 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:29:05.689784 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:05.689881 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:05.690097 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 02:29:05.690245 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":131,"threads_started":1}
I20260619 02:29:05.691293 22133 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0df6294193142d48dcc86ff652627ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34855 } }
I20260619 02:29:05.691496 22133 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:29:05.691555 22133 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0df6294193142d48dcc86ff652627ff, State: Initialized, Role: FOLLOWER
I20260619 02:29:05.691980 22133 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0df6294193142d48dcc86ff652627ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34855 } }
I20260619 02:29:05.692157 22133 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:29:05.692235 22133 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:29:05.692354 22133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:29:05.693265 22133 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0df6294193142d48dcc86ff652627ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34855 } }
I20260619 02:29:05.693656 22133 leader_election.cc:302] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0df6294193142d48dcc86ff652627ff; no voters:
I20260619 02:29:05.693940 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:29:05.693984 22133 leader_election.cc:288] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:29:05.694082 22138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:29:05.694329 22138 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [term 1 LEADER]: Becoming Leader. State: Replica: f0df6294193142d48dcc86ff652627ff, State: Running, Role: LEADER
I20260619 02:29:05.694655 22138 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0df6294193142d48dcc86ff652627ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34855 } }
I20260619 02:29:05.695215 22133 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:05.695830 22138 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0df6294193142d48dcc86ff652627ff. Latest consensus state: current_term: 1 leader_uuid: "f0df6294193142d48dcc86ff652627ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0df6294193142d48dcc86ff652627ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34855 } } }
I20260619 02:29:05.695955 22138 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:05.696257 22140 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0df6294193142d48dcc86ff652627ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0df6294193142d48dcc86ff652627ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34855 } } }
I20260619 02:29:05.696348 22140 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:05.696398 22149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:29:05.699342 22149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:29:05.701936 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:05.696722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45255 (local address 127.0.98.62:34855)
0619 02:29:05.696849 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:05.696854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:05.697463 (+ 609us) server_negotiation.cc:407] Connection header received
0619 02:29:05.697619 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:05.697622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:05.697688 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:05.697742 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:05.698148 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:05.698928 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:05.700087 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:05.700422 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:05.700526 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:05.701442 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:05.701446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:05.701449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:29:05.701460 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:05.701487 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:29:05.701525 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:05.701634 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:05.701756 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 02:29:05.701788 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:29:05.705242 22149 catalog_manager.cc:1347] Generated new cluster ID: 9759bec21e214d1380c031fba6f6f95f
I20260619 02:29:05.705312 22149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:29:05.716466 22149 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:29:05.717293 22149 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:29:05.727116 22149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0df6294193142d48dcc86ff652627ff: Generated new TSK 0
I20260619 02:29:05.727744 22149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:29:05.740095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 22065
W20260619 02:29:05.745442 22050 connection.cc:535] client connection to 127.0.98.62:34855 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34855): Connection reset by peer (error 104)
2026-06-19T02:29:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:29:05 dist-test-slave-xkrt krb5kdc[22164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:29:05 dist-test-slave-xkrt krb5kdc[22164](info): set up 2 sockets
Jun 19 02:29:05 dist-test-slave-xkrt krb5kdc[22164](info): commencing operation
krb5kdc: starting...
W20260619 02:29:06.789865 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:29:06 dist-test-slave-xkrt krb5kdc[22164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:29:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:29:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:29:06.908653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45825
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:29:07.014485 22177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:29:07.014714 22177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:29:07.014753 22177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:29:07.018685 22177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:29:07.018739 22177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:29:07.018759 22177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:29:07.018774 22177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:29:07.018788 22177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:29:07.023928 22177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:29:07.024866 22177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:29:07.025851 22177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:07.034049 22185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:07.034049 22183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:07.034046 22182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:07.034435 22177 server_base.cc:1034] running on GCE node
I20260619 02:29:07.034955 22177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:07.036059 22177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:07.037278 22177 hybrid_clock.cc:648] HybridClock initialized: now 1781836147037255 us; error 38 us; skew 500 ppm
Jun 19 02:29:07 dist-test-slave-xkrt krb5kdc[22164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:07.040094 22177 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:29:07.041007 22177 webserver.cc:460] Webserver started at http://127.0.98.62:40543/ using document root <none> and password file <none>
I20260619 02:29:07.041625 22177 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:07.041724 22177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:07.041934 22177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:07.043984 22177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "9450b6a86c244649a3aa01e0cd45be99"
format_stamp: "Formatted at 2026-06-19 02:29:07 on dist-test-slave-xkrt"
server_key: "a7d37df6a0b770ae721f401e293acb9e"
server_key_iv: "1e75647129bc367ac3034a10af3b06e2"
server_key_version: "encryptionkey@0"
I20260619 02:29:07.044535 22177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "9450b6a86c244649a3aa01e0cd45be99"
format_stamp: "Formatted at 2026-06-19 02:29:07 on dist-test-slave-xkrt"
server_key: "a7d37df6a0b770ae721f401e293acb9e"
server_key_iv: "1e75647129bc367ac3034a10af3b06e2"
server_key_version: "encryptionkey@0"
I20260619 02:29:07.048102 22177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:29:07.050482 22192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:07.051612 22177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:29:07.051748 22177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "9450b6a86c244649a3aa01e0cd45be99"
format_stamp: "Formatted at 2026-06-19 02:29:07 on dist-test-slave-xkrt"
server_key: "a7d37df6a0b770ae721f401e293acb9e"
server_key_iv: "1e75647129bc367ac3034a10af3b06e2"
server_key_version: "encryptionkey@0"
I20260619 02:29:07.051863 22177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:07.056339 22177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:07.059162 22177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:07.059353 22177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:07.066147 22177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45825
I20260619 02:29:07.066157 22244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45825 every 8 connection(s)
I20260619 02:29:07.067022 22177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:29:07.069633 22245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:07.073988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 22177
I20260619 02:29:07.074144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:29:07.074409 392 external_mini_cluster.cc:1428] Setting key 8df957dc8a9d5a8458356a340310e1b4
I20260619 02:29:07.074853 22245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99: Bootstrap starting.
I20260619 02:29:07.077076 22245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:07.077914 22245 log.cc:824] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:07.079450 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:07.075788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60141 (local address 127.0.98.62:45825)
0619 02:29:07.076180 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:07.076204 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:07.076237 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:29:07.076876 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:07.076893 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:07.077209 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:07.077477 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:07.078298 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:07.078302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:07.078313 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:29:07.078344 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:07.078411 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:29:07.078440 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:07.078555 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:07.078660 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 02:29:07.078802 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":121,"threads_started":1}
I20260619 02:29:07.079969 22245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99: No bootstrap required, opened a new log
I20260619 02:29:07.082319 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:29:07.083758 22245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9450b6a86c244649a3aa01e0cd45be99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45825 } }
I20260619 02:29:07.083992 22245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:29:07.084053 22245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9450b6a86c244649a3aa01e0cd45be99, State: Initialized, Role: FOLLOWER
I20260619 02:29:07.084549 22245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9450b6a86c244649a3aa01e0cd45be99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45825 } }
I20260619 02:29:07.084707 22245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:29:07.084800 22245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:29:07.084915 22245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:29:07.085987 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:07.084229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35067 (local address 127.0.98.62:45825)
0619 02:29:07.084350 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:07.084354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:07.084537 (+ 183us) server_negotiation.cc:407] Connection header received
0619 02:29:07.084690 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:07.084693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:07.084732 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:07.084816 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:07.085508 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:07.085511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:07.085514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:29:07.085525 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:07.085577 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:29:07.085651 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:07.085747 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:07.085834 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 02:29:07.085871 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:29:07.085983 22245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9450b6a86c244649a3aa01e0cd45be99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45825 } }
I20260619 02:29:07.086387 22245 leader_election.cc:302] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9450b6a86c244649a3aa01e0cd45be99; no voters:
I20260619 02:29:07.086686 22245 leader_election.cc:288] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:29:07.086778 22255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:29:07.086979 22255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [term 1 LEADER]: Becoming Leader. State: Replica: 9450b6a86c244649a3aa01e0cd45be99, State: Running, Role: LEADER
I20260619 02:29:07.087349 22255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9450b6a86c244649a3aa01e0cd45be99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45825 } }
I20260619 02:29:07.087899 22245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:07.088972 22257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9450b6a86c244649a3aa01e0cd45be99. Latest consensus state: current_term: 1 leader_uuid: "9450b6a86c244649a3aa01e0cd45be99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9450b6a86c244649a3aa01e0cd45be99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45825 } } }
I20260619 02:29:07.089097 22257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:07.089464 22264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:29:07.089425 22256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9450b6a86c244649a3aa01e0cd45be99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9450b6a86c244649a3aa01e0cd45be99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45825 } } }
I20260619 02:29:07.090135 22256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99 [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:07.092593 22264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:29:07.098119 22264 catalog_manager.cc:1347] Generated new cluster ID: df9ee29963874161b1cfc36c3130207f
I20260619 02:29:07.098197 22264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:29:07.113425 22264 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:29:07.113765 22264 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:29:07.122850 22264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9450b6a86c244649a3aa01e0cd45be99: Generated new TSK 0
I20260619 02:29:07.123440 22264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:29:07.131973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 22177
2026-06-19T02:29:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:29:07 dist-test-slave-xkrt krb5kdc[22276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:29:07 dist-test-slave-xkrt krb5kdc[22276](info): set up 2 sockets
Jun 19 02:29:07 dist-test-slave-xkrt krb5kdc[22276](info): commencing operation
krb5kdc: starting...
W20260619 02:29:08.170222 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:29:08 dist-test-slave-xkrt krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:29:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:29:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:29:08.286981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33665
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:29:08.392398 22289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:29:08.392737 22289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:29:08.392817 22289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:29:08.396850 22289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:29:08.396899 22289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:29:08.396919 22289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:29:08.396934 22289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:29:08.396976 22289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:29:08.402112 22289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:29:08.403046 22289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:29:08.403995 22289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:08.412611 22294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:08.412683 22295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:08.412645 22297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:08.412878 22289 server_base.cc:1034] running on GCE node
I20260619 02:29:08.413386 22289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:08.414480 22289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:08.415635 22289 hybrid_clock.cc:648] HybridClock initialized: now 1781836148415604 us; error 44 us; skew 500 ppm
Jun 19 02:29:08 dist-test-slave-xkrt krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:08.418327 22289 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:29:08.419188 22289 webserver.cc:460] Webserver started at http://127.0.98.62:42765/ using document root <none> and password file <none>
I20260619 02:29:08.419708 22289 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:08.419773 22289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:08.419975 22289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:08.421943 22289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "5946717ee74841d5af8d632f2fd216f0"
format_stamp: "Formatted at 2026-06-19 02:29:08 on dist-test-slave-xkrt"
server_key: "5fd26e03c343fdcf6139754e97a041ed"
server_key_iv: "b22b879c991363ceccfa9925071ec8ff"
server_key_version: "encryptionkey@0"
I20260619 02:29:08.422459 22289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "5946717ee74841d5af8d632f2fd216f0"
format_stamp: "Formatted at 2026-06-19 02:29:08 on dist-test-slave-xkrt"
server_key: "5fd26e03c343fdcf6139754e97a041ed"
server_key_iv: "b22b879c991363ceccfa9925071ec8ff"
server_key_version: "encryptionkey@0"
I20260619 02:29:08.425881 22289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:29:08.428421 22304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:08.429613 22289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:29:08.429754 22289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "5946717ee74841d5af8d632f2fd216f0"
format_stamp: "Formatted at 2026-06-19 02:29:08 on dist-test-slave-xkrt"
server_key: "5fd26e03c343fdcf6139754e97a041ed"
server_key_iv: "b22b879c991363ceccfa9925071ec8ff"
server_key_version: "encryptionkey@0"
I20260619 02:29:08.429848 22289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:08.456810 22289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:08.459549 22289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:08.459741 22289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:08.466591 22289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33665
I20260619 02:29:08.466650 22356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33665 every 8 connection(s)
I20260619 02:29:08.467586 22289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:29:08.470194 22357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:08.472517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 22289
I20260619 02:29:08.472666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:29:08.472947 392 external_mini_cluster.cc:1428] Setting key 75f84429e969d7e54b135f64bd8a6bc7
I20260619 02:29:08.476243 22357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0: Bootstrap starting.
I20260619 02:29:08.478682 22357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:08.479429 22357 log.cc:824] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:08.481647 22357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0: No bootstrap required, opened a new log
Jun 19 02:29:08 dist-test-slave-xkrt krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:29:08.484949 22357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5946717ee74841d5af8d632f2fd216f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33665 } }
I20260619 02:29:08.485159 22357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:29:08.485221 22357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5946717ee74841d5af8d632f2fd216f0, State: Initialized, Role: FOLLOWER
I20260619 02:29:08.485700 22357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5946717ee74841d5af8d632f2fd216f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33665 } }
I20260619 02:29:08.485859 22357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:29:08.485980 22357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:29:08.486129 22357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:29:08.486872 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:08.474261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58453 (local address 127.0.98.62:33665)
0619 02:29:08.474635 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:08.474656 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:08.474689 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:29:08.475311 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:08.475326 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:08.475636 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:08.475906 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:08.476790 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:08.477807 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:08.478953 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:08.479395 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:08.479510 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:08.482355 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:08.482360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:08.482372 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:08.482396 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:08.484051 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:08.484494 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:08.484499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:08.484503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:08.484604 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:08.484886 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:08.484889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:08.484891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:08.485343 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:08.485544 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:08.485814 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 02:29:08.486126 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":124,"threads_started":1}
I20260619 02:29:08.487026 22357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5946717ee74841d5af8d632f2fd216f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33665 } }
I20260619 02:29:08.487356 22357 leader_election.cc:302] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5946717ee74841d5af8d632f2fd216f0; no voters:
I20260619 02:29:08.487663 22357 leader_election.cc:288] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:29:08.487725 22362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:29:08.488894 22357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:08.488682 22362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [term 1 LEADER]: Becoming Leader. State: Replica: 5946717ee74841d5af8d632f2fd216f0, State: Running, Role: LEADER
I20260619 02:29:08.489234 22362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5946717ee74841d5af8d632f2fd216f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33665 } }
I20260619 02:29:08.489668 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:29:08.490774 22363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5946717ee74841d5af8d632f2fd216f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5946717ee74841d5af8d632f2fd216f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33665 } } }
I20260619 02:29:08.491170 22363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:08.491052 22364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5946717ee74841d5af8d632f2fd216f0. Latest consensus state: current_term: 1 leader_uuid: "5946717ee74841d5af8d632f2fd216f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5946717ee74841d5af8d632f2fd216f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33665 } } }
I20260619 02:29:08.491505 22364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:08.492429 22374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:29:08.495436 22374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:29:08.500873 22374 catalog_manager.cc:1347] Generated new cluster ID: 92af55ff8aef48c79ecefb605ca96a97
I20260619 02:29:08.500958 22374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:29:08.501135 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:08.491820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54785 (local address 127.0.98.62:33665)
0619 02:29:08.491925 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:08.491929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:08.492360 (+ 431us) server_negotiation.cc:407] Connection header received
0619 02:29:08.492535 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:08.492538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:08.492582 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:08.492630 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:08.493420 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:08.494227 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:08.495344 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:08.495656 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:08.495759 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:08.497343 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:08.497346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:08.497349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:08.497372 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:08.499009 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:08.499553 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:08.499557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:08.499558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:08.499645 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:08.499943 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:08.499946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:08.499948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:08.500147 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:08.500278 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:08.500859 (+ 581us) server_negotiation.cc:299] Negotiation successful
0619 02:29:08.500968 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:29:08.522997 22374 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:29:08.524240 22374 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:29:08.531734 22374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5946717ee74841d5af8d632f2fd216f0: Generated new TSK 0
I20260619 02:29:08.532497 22374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:29:08.536612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 22289
2026-06-19T02:29:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:29:08 dist-test-slave-xkrt krb5kdc[22388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:29:08 dist-test-slave-xkrt krb5kdc[22388](info): set up 2 sockets
Jun 19 02:29:08 dist-test-slave-xkrt krb5kdc[22388](info): commencing operation
krb5kdc: starting...
W20260619 02:29:09.577886 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:29:09 dist-test-slave-xkrt krb5kdc[22388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:29:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:29:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:29:09.698458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46269
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:29:09.804966 22401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:29:09.805248 22401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:29:09.805326 22401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:29:09.809230 22401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:29:09.809312 22401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:29:09.809350 22401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:29:09.809407 22401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:29:09.809446 22401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:29:09.814569 22401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:29:09.815550 22401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:29:09.816522 22401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:09.824894 22406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:09.824918 22409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:09.824951 22407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:09.825103 22401 server_base.cc:1034] running on GCE node
I20260619 02:29:09.825855 22401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:09.827000 22401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:09.828191 22401 hybrid_clock.cc:648] HybridClock initialized: now 1781836149828161 us; error 52 us; skew 500 ppm
Jun 19 02:29:09 dist-test-slave-xkrt krb5kdc[22388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:09.830969 22401 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:29:09.831874 22401 webserver.cc:460] Webserver started at http://127.0.98.62:32899/ using document root <none> and password file <none>
I20260619 02:29:09.832448 22401 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:09.832540 22401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:09.832769 22401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:09.834906 22401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "8dc66242fc1b4249ac103a5a79e99043"
format_stamp: "Formatted at 2026-06-19 02:29:09 on dist-test-slave-xkrt"
server_key: "47230c33d2c420eb593b7b2c7df0b8f6"
server_key_iv: "2f3bb855f35071c40106de2b003e846d"
server_key_version: "encryptionkey@0"
I20260619 02:29:09.835481 22401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "8dc66242fc1b4249ac103a5a79e99043"
format_stamp: "Formatted at 2026-06-19 02:29:09 on dist-test-slave-xkrt"
server_key: "47230c33d2c420eb593b7b2c7df0b8f6"
server_key_iv: "2f3bb855f35071c40106de2b003e846d"
server_key_version: "encryptionkey@0"
I20260619 02:29:09.839166 22401 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:29:09.841634 22416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:09.842830 22401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:29:09.842964 22401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "8dc66242fc1b4249ac103a5a79e99043"
format_stamp: "Formatted at 2026-06-19 02:29:09 on dist-test-slave-xkrt"
server_key: "47230c33d2c420eb593b7b2c7df0b8f6"
server_key_iv: "2f3bb855f35071c40106de2b003e846d"
server_key_version: "encryptionkey@0"
I20260619 02:29:09.843088 22401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:09.862859 22401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:09.865769 22401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:09.865978 22401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:09.872831 22401 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46269
I20260619 02:29:09.872845 22468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46269 every 8 connection(s)
I20260619 02:29:09.873798 22401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:29:09.876462 22469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:09.881747 22469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043: Bootstrap starting.
I20260619 02:29:09.883956 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 22401
I20260619 02:29:09.884090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:29:09.884253 22469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:09.884393 392 external_mini_cluster.cc:1428] Setting key 6d092619f8ee0ac17311510657da92dc
I20260619 02:29:09.885083 22469 log.cc:824] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:09.887326 22469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043: No bootstrap required, opened a new log
I20260619 02:29:09.891554 22469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc66242fc1b4249ac103a5a79e99043" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46269 } }
I20260619 02:29:09.891796 22469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:29:09.891866 22469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc66242fc1b4249ac103a5a79e99043, State: Initialized, Role: FOLLOWER
I20260619 02:29:09.892354 22469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc66242fc1b4249ac103a5a79e99043" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46269 } }
I20260619 02:29:09.892521 22469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:29:09.892577 22469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:29:09.892670 22469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:29:09.893800 22469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc66242fc1b4249ac103a5a79e99043" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46269 } }
I20260619 02:29:09.894116 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:09.885521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35401 (local address 127.0.98.62:46269)
0619 02:29:09.885966 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:09.885986 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:09.886030 (+ 44us) server_negotiation.cc:407] Connection header received
0619 02:29:09.886846 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:09.886868 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:09.887302 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:09.887653 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:09.888062 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:09.889523 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:09.890402 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:09.891084 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:09.891253 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:09.892086 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:09.892094 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:09.892109 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:29:09.892592 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:09.892660 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:29:09.892694 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:09.892793 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:09.892979 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 02:29:09.893169 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
I20260619 02:29:09.894134 22469 leader_election.cc:302] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dc66242fc1b4249ac103a5a79e99043; no voters:
I20260619 02:29:09.894428 22469 leader_election.cc:288] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:29:09.894598 22474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:29:09.894897 22474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [term 1 LEADER]: Becoming Leader. State: Replica: 8dc66242fc1b4249ac103a5a79e99043, State: Running, Role: LEADER
I20260619 02:29:09.895241 22474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc66242fc1b4249ac103a5a79e99043" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46269 } }
I20260619 02:29:09.895599 22469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:09.896450 22474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dc66242fc1b4249ac103a5a79e99043. Latest consensus state: current_term: 1 leader_uuid: "8dc66242fc1b4249ac103a5a79e99043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc66242fc1b4249ac103a5a79e99043" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46269 } } }
I20260619 02:29:09.896648 22474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:09.896935 22475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dc66242fc1b4249ac103a5a79e99043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc66242fc1b4249ac103a5a79e99043" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46269 } } }
I20260619 02:29:09.897097 22475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043 [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:09.897166 22483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:29:09.897166 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:29:09.901167 22483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:29:09.904930 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:09.899831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54369 (local address 127.0.98.62:46269)
0619 02:29:09.899943 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:09.899950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:09.899964 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:29:09.900001 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:09.900004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:09.900062 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:09.900153 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:09.900445 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:09.901831 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:09.903004 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:09.903312 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:09.903449 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:09.904354 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:09.904358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:09.904361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:29:09.904375 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:09.904408 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:29:09.904420 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:09.904506 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:09.904605 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 02:29:09.904679 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:29:09.907469 22483 catalog_manager.cc:1347] Generated new cluster ID: cc6588b99af14dedbfb583f87f1e554d
I20260619 02:29:09.907553 22483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:29:09.915691 22483 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:29:09.916616 22483 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:29:09.921452 22483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dc66242fc1b4249ac103a5a79e99043: Generated new TSK 0
I20260619 02:29:09.922123 22483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:29:09.930544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 22401
2026-06-19T02:29:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:29:09 dist-test-slave-xkrt krb5kdc[22500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:29:09 dist-test-slave-xkrt krb5kdc[22500](info): set up 2 sockets
Jun 19 02:29:09 dist-test-slave-xkrt krb5kdc[22500](info): commencing operation
krb5kdc: starting...
W20260619 02:29:10.957919 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:29:11 dist-test-slave-xkrt krb5kdc[22500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:29:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:29:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:29:11.076397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46291
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:29:11.182016 22513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:29:11.182307 22513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:29:11.182379 22513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:29:11.186352 22513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:29:11.186447 22513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:29:11.186487 22513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:29:11.186546 22513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:29:11.186585 22513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:29:11.191782 22513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:29:11.192826 22513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:29:11.193843 22513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:11.201862 22519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:11.201865 22518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:11.201915 22521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:11.202086 22513 server_base.cc:1034] running on GCE node
I20260619 02:29:11.202678 22513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:11.203733 22513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:11.204892 22513 hybrid_clock.cc:648] HybridClock initialized: now 1781836151204883 us; error 25 us; skew 500 ppm
Jun 19 02:29:11 dist-test-slave-xkrt krb5kdc[22500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:11.207553 22513 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:29:11.208352 22513 webserver.cc:460] Webserver started at http://127.0.98.62:40923/ using document root <none> and password file <none>
I20260619 02:29:11.208839 22513 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:11.208892 22513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:11.209043 22513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:11.210988 22513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "d1c6cb01637e4920a493a84fece4bf1e"
format_stamp: "Formatted at 2026-06-19 02:29:11 on dist-test-slave-xkrt"
server_key: "d77c0cc050fcdac3b805ecd24046f3de"
server_key_iv: "2cdd7032b168d69cd65fcf4bc866d5bf"
server_key_version: "encryptionkey@0"
I20260619 02:29:11.211555 22513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "d1c6cb01637e4920a493a84fece4bf1e"
format_stamp: "Formatted at 2026-06-19 02:29:11 on dist-test-slave-xkrt"
server_key: "d77c0cc050fcdac3b805ecd24046f3de"
server_key_iv: "2cdd7032b168d69cd65fcf4bc866d5bf"
server_key_version: "encryptionkey@0"
I20260619 02:29:11.215147 22513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 02:29:11.217437 22528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:11.218737 22513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:29:11.218880 22513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "d1c6cb01637e4920a493a84fece4bf1e"
format_stamp: "Formatted at 2026-06-19 02:29:11 on dist-test-slave-xkrt"
server_key: "d77c0cc050fcdac3b805ecd24046f3de"
server_key_iv: "2cdd7032b168d69cd65fcf4bc866d5bf"
server_key_version: "encryptionkey@0"
I20260619 02:29:11.218990 22513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:11.223585 22513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:11.226509 22513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:11.226711 22513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:11.233024 22513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46291
I20260619 02:29:11.233078 22580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46291 every 8 connection(s)
I20260619 02:29:11.233958 22513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:29:11.236588 22581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:11.241842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 22513
I20260619 02:29:11.241978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:29:11.241986 22581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e: Bootstrap starting.
I20260619 02:29:11.242259 392 external_mini_cluster.cc:1428] Setting key fd5626ea7ad6f0e9922fc6f86a6cd9f4
I20260619 02:29:11.244333 22581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:11.245040 22581 log.cc:824] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:11.247172 22581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e: No bootstrap required, opened a new log
I20260619 02:29:11.247304 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:11.243594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59561 (local address 127.0.98.62:46291)
0619 02:29:11.243997 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:11.244012 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:11.244045 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:29:11.244697 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:11.244715 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:11.244983 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:11.245342 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:11.246133 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:11.246137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:11.246147 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:29:11.246156 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:11.246201 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:29:11.246219 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:11.246301 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:11.246429 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 02:29:11.246589 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":125,"threads_started":1}
I20260619 02:29:11.250298 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:29:11.250267 22581 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c6cb01637e4920a493a84fece4bf1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46291 } }
I20260619 02:29:11.250519 22581 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:29:11.250567 22581 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c6cb01637e4920a493a84fece4bf1e, State: Initialized, Role: FOLLOWER
I20260619 02:29:11.251021 22581 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c6cb01637e4920a493a84fece4bf1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46291 } }
I20260619 02:29:11.251196 22581 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:29:11.251262 22581 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:29:11.251360 22581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:29:11.252468 22581 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c6cb01637e4920a493a84fece4bf1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46291 } }
I20260619 02:29:11.252825 22581 leader_election.cc:302] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1c6cb01637e4920a493a84fece4bf1e; no voters:
I20260619 02:29:11.253158 22581 leader_election.cc:288] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:29:11.253259 22591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:29:11.253417 22591 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [term 1 LEADER]: Becoming Leader. State: Replica: d1c6cb01637e4920a493a84fece4bf1e, State: Running, Role: LEADER
I20260619 02:29:11.253748 22591 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c6cb01637e4920a493a84fece4bf1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46291 } }
I20260619 02:29:11.254472 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:11.252514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53057 (local address 127.0.98.62:46291)
0619 02:29:11.252671 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:11.252676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:11.252914 (+ 238us) server_negotiation.cc:407] Connection header received
0619 02:29:11.253085 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:11.253087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:11.253116 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:11.253212 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:11.253935 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:11.253940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:11.253943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:29:11.253954 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:11.253977 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:29:11.253988 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:11.254082 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:11.254338 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 02:29:11.254376 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 02:29:11.254662 22581 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:11.255437 22593 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1c6cb01637e4920a493a84fece4bf1e. Latest consensus state: current_term: 1 leader_uuid: "d1c6cb01637e4920a493a84fece4bf1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c6cb01637e4920a493a84fece4bf1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46291 } } }
I20260619 02:29:11.255486 22592 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1c6cb01637e4920a493a84fece4bf1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c6cb01637e4920a493a84fece4bf1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46291 } } }
I20260619 02:29:11.255578 22593 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:11.255618 22592 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:11.255919 22600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:29:11.258785 22600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:29:11.263594 22600 catalog_manager.cc:1347] Generated new cluster ID: cdb44334b20f4ae99c73a2d0d8a1948a
I20260619 02:29:11.263657 22600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:29:11.276587 22600 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:29:11.276876 22600 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:29:11.287012 22600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1c6cb01637e4920a493a84fece4bf1e: Generated new TSK 0
I20260619 02:29:11.287719 22600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:29:11.297099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 22513
2026-06-19T02:29:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1367 ms)
[----------] 18 tests from AuthTokenIssuingTest (25189 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:29:11 dist-test-slave-xkrt krb5kdc[22612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:29:11 dist-test-slave-xkrt krb5kdc[22612](info): set up 2 sockets
Jun 19 02:29:11 dist-test-slave-xkrt krb5kdc[22612](info): commencing operation
krb5kdc: starting...
W20260619 02:29:12.325990 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:29:12 dist-test-slave-xkrt krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:29:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:29:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:29:12.444408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36569
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36569,127.0.98.61:40709,127.0.98.60:38111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:29:12.555327 22625 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:29:12.555589 22625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:29:12.555613 22625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:29:12.555651 22625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:29:12.559579 22625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:29:12.559630 22625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:29:12.559650 22625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:29:12.559666 22625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:29:12.559681 22625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:29:12.564829 22625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36569,127.0.98.61:40709,127.0.98.60:38111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:29:12.565832 22625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:29:12.566776 22625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:12.575155 22633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:12.575138 22630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:12.575189 22625 server_base.cc:1034] running on GCE node
W20260619 02:29:12.575138 22631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:12.575994 22625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:12.577047 22625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:12.578228 22625 hybrid_clock.cc:648] HybridClock initialized: now 1781836152578217 us; error 32 us; skew 500 ppm
Jun 19 02:29:12 dist-test-slave-xkrt krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:12.583892 22625 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:29:12.584743 22625 webserver.cc:460] Webserver started at http://127.0.98.62:34237/ using document root <none> and password file <none>
I20260619 02:29:12.585256 22625 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:12.585310 22625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:12.585467 22625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:12.587484 22625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "a0eb5f5b151b461aa93848ddb5d726c1"
format_stamp: "Formatted at 2026-06-19 02:29:12 on dist-test-slave-xkrt"
server_key: "97ec1eff7425976264e1f60f3cd8e8a7"
server_key_iv: "0e21b4bda5e83ce10255eb091737d109"
server_key_version: "encryptionkey@0"
I20260619 02:29:12.587968 22625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0eb5f5b151b461aa93848ddb5d726c1"
format_stamp: "Formatted at 2026-06-19 02:29:12 on dist-test-slave-xkrt"
server_key: "97ec1eff7425976264e1f60f3cd8e8a7"
server_key_iv: "0e21b4bda5e83ce10255eb091737d109"
server_key_version: "encryptionkey@0"
I20260619 02:29:12.591537 22625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:29:12.593837 22640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:12.595028 22625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:29:12.595166 22625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "a0eb5f5b151b461aa93848ddb5d726c1"
format_stamp: "Formatted at 2026-06-19 02:29:12 on dist-test-slave-xkrt"
server_key: "97ec1eff7425976264e1f60f3cd8e8a7"
server_key_iv: "0e21b4bda5e83ce10255eb091737d109"
server_key_version: "encryptionkey@0"
I20260619 02:29:12.595264 22625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:12.620100 22625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:12.622958 22625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:12.623162 22625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:12.629599 22625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36569
I20260619 02:29:12.629858 22692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36569 every 8 connection(s)
I20260619 02:29:12.630575 22625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:29:12.633209 22693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:12.636936 22693 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:29:12.640321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 22625
I20260619 02:29:12.640456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:29:12.640736 392 external_mini_cluster.cc:1428] Setting key bdc634d55e0fbd484ecbdc2516f2c28d
Jun 19 02:29:12 dist-test-slave-xkrt krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260619 02:29:12.652645 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.639189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.0.98.62:36569)
0619 02:29:12.639605 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:12.639623 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:12.639892 (+ 269us) server_negotiation.cc:407] Connection header received
0619 02:29:12.641138 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:12.641142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:12.641313 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:12.641459 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:12.642594 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.643465 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.644488 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.644846 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.645008 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:12.647301 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:12.647312 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:12.647322 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:12.647341 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:12.649062 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:12.649533 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:12.649588 (+ 55us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:12.649590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:12.649746 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:12.650063 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:12.650066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:12.650068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:12.650651 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:12.650786 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:12.651331 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 02:29:12.651599 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":278,"thread_start_us":147,"threads_started":1}
I20260619 02:29:12.652645 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.639188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36569 (local address 127.0.0.1:46384)
0619 02:29:12.639742 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.639789 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:12.640848 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:12.641518 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:12.641526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:12.642043 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:12.642435 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:12.642456 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.643588 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.643591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:12.644367 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:12.644377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.644994 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.644998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:12.645105 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:12.645728 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:12.645744 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:12.647158 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:12.649184 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:12.649193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:12.649202 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:12.649424 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:12.649843 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:12.649847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:12.649849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:12.649965 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:12.650768 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:12.650774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:12.650992 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 02:29:12.651212 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 02:29:12.651654 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":140,"spinlock_wait_cycles":61184,"thread_start_us":161,"threads_started":1}
I20260619 02:29:12.655021 22693 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:29:12.655762 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.655510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:48612)
0619 02:29:12.655644 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.655707 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 02:29:12.655977 22644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
I20260619 02:29:12.656786 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.656599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:48624)
0619 02:29:12.656697 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.656739 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 02:29:12.657065 22693 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40709: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 02:29:12.666618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40709
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36569,127.0.98.61:40709,127.0.98.60:38111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:29:12.706319 22693 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } attempt: 1
I20260619 02:29:12.707278 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.706976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:48626)
0619 02:29:12.707138 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.707201 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 02:29:12.708052 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.707836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:48642)
0619 02:29:12.707945 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.707997 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:29:12.708230 22693 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40709: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
W20260619 02:29:12.774489 22700 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:29:12.774745 22700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:29:12.774768 22700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:29:12.774859 22700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 02:29:12.776407 22693 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } attempt: 2
I20260619 02:29:12.777315 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.776974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:48656)
0619 02:29:12.777150 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.777224 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 02:29:12.778179 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.777918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:48672)
0619 02:29:12.778023 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.778119 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 02:29:12.778301 22693 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40709: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
W20260619 02:29:12.780283 22700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:29:12.780332 22700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:29:12.780351 22700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:29:12.780400 22700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:29:12.780429 22700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:29:12.785825 22700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36569,127.0.98.61:40709,127.0.98.60:38111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:29:12.786875 22700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:29:12.787873 22700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:12.796293 22706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:12.796285 22708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:12.796307 22705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:12.796344 22700 server_base.cc:1034] running on GCE node
I20260619 02:29:12.797091 22700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:12.798296 22700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:12.799497 22700 hybrid_clock.cc:648] HybridClock initialized: now 1781836152799440 us; error 41 us; skew 500 ppm
Jun 19 02:29:12 dist-test-slave-xkrt krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:12.802544 22700 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 02:29:12.803356 22700 webserver.cc:460] Webserver started at http://127.0.98.61:44501/ using document root <none> and password file <none>
I20260619 02:29:12.803943 22700 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:12.804032 22700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:12.804247 22700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:12.806316 22700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/data/instance:
uuid: "85bbc9b3b2914d55be8e315d1a0cfedf"
format_stamp: "Formatted at 2026-06-19 02:29:12 on dist-test-slave-xkrt"
server_key: "064d5b076ab3cd3cbee8e7ff009c808e"
server_key_iv: "a53360c1f292d81ccf2ef84850d1e142"
server_key_version: "encryptionkey@0"
I20260619 02:29:12.806895 22700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/wal/instance:
uuid: "85bbc9b3b2914d55be8e315d1a0cfedf"
format_stamp: "Formatted at 2026-06-19 02:29:12 on dist-test-slave-xkrt"
server_key: "064d5b076ab3cd3cbee8e7ff009c808e"
server_key_iv: "a53360c1f292d81ccf2ef84850d1e142"
server_key_version: "encryptionkey@0"
I20260619 02:29:12.810741 22700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:29:12.813067 22715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:12.814249 22700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:29:12.814381 22700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/wal
uuid: "85bbc9b3b2914d55be8e315d1a0cfedf"
format_stamp: "Formatted at 2026-06-19 02:29:12 on dist-test-slave-xkrt"
server_key: "064d5b076ab3cd3cbee8e7ff009c808e"
server_key_iv: "a53360c1f292d81ccf2ef84850d1e142"
server_key_version: "encryptionkey@0"
I20260619 02:29:12.814486 22700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:12.841813 22700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:12.844760 22700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:12.844975 22700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:12.851732 22700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40709
I20260619 02:29:12.852023 22767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40709 every 8 connection(s)
I20260619 02:29:12.852749 22700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/data/info.pb
I20260619 02:29:12.855353 22768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:12.858949 22768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:29:12.862924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 22700
I20260619 02:29:12.863065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-1/wal/instance
I20260619 02:29:12.863318 392 external_mini_cluster.cc:1428] Setting key 2c67712d4099e71694c2cdd52ab6aaa4
Jun 19 02:29:12 dist-test-slave-xkrt krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:29:12.869498 22693 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } attempt: 3
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 02:29:12.872797 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.860742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46390 (local address 127.0.98.62:36569)
0619 02:29:12.860882 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:12.860885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:12.861671 (+ 786us) server_negotiation.cc:407] Connection header received
0619 02:29:12.862580 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:12.862583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:12.862639 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:12.862746 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:12.864007 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.864540 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.865673 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.865913 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.866031 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:12.868696 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:12.868700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:12.868705 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:12.868738 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:12.870573 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:12.871245 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:12.871249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:12.871254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:12.871342 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:12.871688 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:12.871690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:12.871691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:12.871850 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:12.871972 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:12.872527 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 02:29:12.872644 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260619 02:29:12.873775 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.861045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36569 (local address 127.0.0.1:46390)
0619 02:29:12.861466 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.861516 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:12.862385 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:12.862925 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:12.862932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:12.863352 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:12.863863 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:12.863875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.864662 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.864666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:12.865508 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:12.865517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.866063 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.866066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:12.866234 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:12.866814 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:12.866829 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:12.868512 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:12.870696 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:12.870702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:12.870716 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:12.871135 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:12.871431 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:12.871434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:12.871436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:12.871593 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:12.871972 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:12.871977 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:12.872253 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 02:29:12.872446 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 02:29:12.872740 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260619 02:29:12.875247 22768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:29:12 dist-test-slave-xkrt krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 02:29:12 dist-test-slave-xkrt krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 02:29:12.880717 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.869963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:48674)
0619 02:29:12.870120 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.870135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:12.870211 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:12.870904 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:12.870906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:12.871082 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:12.871169 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:12.871174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.872553 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.872556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:12.873330 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:12.873341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.874163 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.874167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:12.874273 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:12.874750 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:12.874763 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:12.876983 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:12.878877 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:12.878882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:12.878886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:12.879158 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:12.879525 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:12.879529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:12.879530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:12.879619 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:12.880189 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:12.880193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:12.880293 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 02:29:12.880383 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 02:29:12.880525 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 02:29:12.881953 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.870181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48674 (local address 127.0.98.61:40709)
0619 02:29:12.870519 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:12.870536 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:12.870559 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:29:12.870602 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:12.870610 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:12.870786 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:12.870920 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:12.871288 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.872419 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.873477 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.874009 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.874162 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:12.877112 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:12.877121 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:12.877133 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:12.877155 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:12.878785 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:12.879299 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:12.879307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:12.879311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:12.879407 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:12.879751 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:12.879754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:12.879757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:12.880057 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:12.880931 (+ 874us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:12.881064 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 02:29:12.881263 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"spinlock_wait_cycles":1792,"thread_start_us":135,"threads_started":1}
I20260619 02:29:12.883720 22693 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:29:12.884251 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.875710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:48690)
0619 02:29:12.875795 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.875809 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:12.875873 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:12.876308 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:12.876311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:12.876472 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:12.876547 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:12.876551 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.877724 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.877730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:12.878666 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:12.878680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.879182 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.879185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:12.879279 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:12.879693 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:12.879704 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:12.880822 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:12.882772 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:12.882775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:12.882777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:12.882990 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:12.883263 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:12.883265 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:12.883267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:12.883344 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:12.883763 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:12.883765 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:12.883845 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 02:29:12.883959 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 02:29:12.884086 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 02:29:12.884295 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.875818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48690 (local address 127.0.98.61:40709)
0619 02:29:12.876131 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:12.876137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:12.876147 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:29:12.876173 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:12.876175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:12.876204 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:12.876300 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:12.876694 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.877302 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.878795 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:12.879081 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:12.879185 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:12.881001 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:12.881007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:12.881012 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:12.881064 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:12.882654 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:12.883097 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:12.883100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:12.883101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:12.883169 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:12.883438 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:12.883444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:12.883446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:12.883668 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:12.883772 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:12.884002 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 02:29:12.884140 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":252,"spinlock_wait_cycles":768,"thread_start_us":150,"threads_started":1}
I20260619 02:29:12.884454 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.884205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:56912)
0619 02:29:12.884307 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.884368 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 02:29:12.885075 22768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:29:12.885288 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.885050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:56918)
0619 02:29:12.885179 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.885228 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 02:29:12.885628 22693 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38111: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
I20260619 02:29:12.885756 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.885476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:56920)
0619 02:29:12.885603 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.885674 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 02:29:12.885908 22717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
I20260619 02:29:12.886695 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.886464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:56932)
0619 02:29:12.886597 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.886642 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 02:29:12.886873 22768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38111: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 02:29:12.890872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38111
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36569,127.0.98.61:40709,127.0.98.60:38111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:29:12.916816 22693 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } attempt: 1
I20260619 02:29:12.917732 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.917369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:56942)
0619 02:29:12.917566 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.917637 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 02:29:12.918529 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.918286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:56956)
0619 02:29:12.918421 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.918472 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260619 02:29:12.918794 22693 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38111: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
I20260619 02:29:12.936092 22768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } attempt: 1
I20260619 02:29:12.936939 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.936614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:56972)
0619 02:29:12.936777 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.936847 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 02:29:12.937686 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.937440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:56988)
0619 02:29:12.937575 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.937626 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 02:29:12.937834 22768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38111: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
I20260619 02:29:12.994019 22693 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } attempt: 2
I20260619 02:29:12.994885 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.994600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:57004)
0619 02:29:12.994748 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.994808 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 02:29:12.995822 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:12.995576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:57018)
0619 02:29:12.995713 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:12.995758 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260619 02:29:12.995988 22693 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38111: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
W20260619 02:29:13.000406 22776 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:29:13.000829 22776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:29:13.000886 22776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:29:13.000970 22776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:29:13.005110 22776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:29:13.005169 22776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:29:13.005189 22776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:29:13.005235 22776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:29:13.005268 22776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:29:13.005985 22768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } attempt: 2
I20260619 02:29:13.006744 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.006430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:57028)
0619 02:29:13.006601 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:13.006663 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 02:29:13.007458 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.007247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:57040)
0619 02:29:13.007354 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:13.007401 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 02:29:13.007613 22768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38111: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38111: connect: Connection refused (error 111)
I20260619 02:29:13.011045 22776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36569,127.0.98.61:40709,127.0.98.60:38111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:29:13.012169 22776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:29:13.013212 22776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:13.021824 22784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:13.021828 22781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:13.021828 22782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:13.022123 22776 server_base.cc:1034] running on GCE node
I20260619 02:29:13.022753 22776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:13.023943 22776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:13.025153 22776 hybrid_clock.cc:648] HybridClock initialized: now 1781836153025138 us; error 33 us; skew 500 ppm
Jun 19 02:29:13 dist-test-slave-xkrt krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:13.028386 22776 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 02:29:13.029393 22776 webserver.cc:460] Webserver started at http://127.0.98.60:44871/ using document root <none> and password file <none>
I20260619 02:29:13.030131 22776 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:13.030229 22776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:13.030469 22776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:13.032501 22776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/data/instance:
uuid: "784538c8aff549d0bcac7937ee08e0c9"
format_stamp: "Formatted at 2026-06-19 02:29:13 on dist-test-slave-xkrt"
server_key: "2c58d526043363d399cdab06a6ae7ded"
server_key_iv: "8ab3a3b0dada7160d8b361fad784b860"
server_key_version: "encryptionkey@0"
I20260619 02:29:13.033074 22776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/wal/instance:
uuid: "784538c8aff549d0bcac7937ee08e0c9"
format_stamp: "Formatted at 2026-06-19 02:29:13 on dist-test-slave-xkrt"
server_key: "2c58d526043363d399cdab06a6ae7ded"
server_key_iv: "8ab3a3b0dada7160d8b361fad784b860"
server_key_version: "encryptionkey@0"
I20260619 02:29:13.036787 22776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:29:13.039227 22791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:13.040426 22776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:29:13.040686 22776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/wal
uuid: "784538c8aff549d0bcac7937ee08e0c9"
format_stamp: "Formatted at 2026-06-19 02:29:13 on dist-test-slave-xkrt"
server_key: "2c58d526043363d399cdab06a6ae7ded"
server_key_iv: "8ab3a3b0dada7160d8b361fad784b860"
server_key_version: "encryptionkey@0"
I20260619 02:29:13.040804 22776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:13.053776 22776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:13.057035 22776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:13.057245 22776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:13.064606 22776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38111
I20260619 02:29:13.064899 22843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38111 every 8 connection(s)
I20260619 02:29:13.065621 22776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/data/info.pb
I20260619 02:29:13.066443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 22776
I20260619 02:29:13.066577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781835873883564-392-0/minicluster-data/master-2/wal/instance
I20260619 02:29:13.066871 392 external_mini_cluster.cc:1428] Setting key 0672ff0c2e1949f9b3e7812c8c8457c7
I20260619 02:29:13.068909 22844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:13.073405 22844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:29:13 dist-test-slave-xkrt krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 02:29:13 dist-test-slave-xkrt krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:29:13.081794 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.068208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.0.98.60:38111)
0619 02:29:13.068603 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.068630 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.068665 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:29:13.069362 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.069381 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.069774 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.070055 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.070950 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.071992 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.073222 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.073787 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.073992 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.076866 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:13.076872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:13.076890 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:13.076915 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:13.078657 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.079158 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.079179 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.079185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.079334 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.079613 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.079617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.079619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.080082 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:13.080273 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:13.080581 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 02:29:13.080837 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":135,"threads_started":1}
I20260619 02:29:13.084398 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.075282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.0.98.62:36569)
0619 02:29:13.075406 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.075410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.075782 (+ 372us) server_negotiation.cc:407] Connection header received
0619 02:29:13.075957 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.075960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.076014 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.076103 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.076839 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.077328 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.078277 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.078498 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.078613 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.080940 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:13.080943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:13.080945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:13.080967 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:13.082574 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.083008 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.083011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.083013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.083121 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.083387 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.083391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.083394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.083662 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:13.083776 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:13.084169 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 02:29:13.084286 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:29:13.084600 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.075229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36569 (local address 127.0.0.1:46392)
0619 02:29:13.075658 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:13.075696 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:13.075841 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:13.076151 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:13.076156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:13.076582 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:13.076705 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:13.076711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.077443 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.077447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:13.078145 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:13.078154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.078613 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.078616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:13.078737 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.079245 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:13.079258 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:13.080815 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:13.082677 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:13.082684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:13.082690 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:13.082911 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:13.083217 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:13.083219 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:13.083220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:13.083295 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:13.083773 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:13.083801 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:13.083939 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 02:29:13.084112 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 02:29:13.084314 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":180,"threads_started":1}
I20260619 02:29:13.085896 22844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:29:13 dist-test-slave-xkrt krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 02:29:13 dist-test-slave-xkrt krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 02:29:13.094805 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.084917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.0.98.62:36569)
0619 02:29:13.085019 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.085023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.085260 (+ 237us) server_negotiation.cc:407] Connection header received
0619 02:29:13.085381 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.085385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.085427 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.085506 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.086262 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.087040 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.088157 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.088506 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.088636 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.091224 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:13.091230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:13.091236 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:13.091269 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:13.092955 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.093422 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.093426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.093428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.093514 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.093837 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.093843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.093847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.094078 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:13.094267 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:13.094499 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 02:29:13.094627 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:29:13.095144 22693 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } attempt: 3
I20260619 02:29:13.095542 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.086383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:48696)
0619 02:29:13.086537 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:13.086550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:13.086618 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:13.086832 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:13.086835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:13.086989 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:13.087126 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:13.087135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.088191 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.088194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:13.089057 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:13.089070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.089744 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.089747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:13.089819 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.090828 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:13.090839 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:13.091932 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:13.093779 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:13.093781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:13.093783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:13.094027 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:13.094392 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:13.094394 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:13.094395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:13.094509 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:13.095009 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:13.095012 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:13.095115 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 02:29:13.095231 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 02:29:13.095368 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 02:29:13.095709 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.086478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.0.98.61:40709)
0619 02:29:13.086609 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.086619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.086631 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:29:13.086694 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.086696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.086734 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.086795 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.087242 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.088064 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.089211 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.089621 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.089763 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.092063 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:13.092066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:13.092069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:13.092099 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:13.093684 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.094175 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.094183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.094185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.094286 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.094617 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.094622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.094624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.094847 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:13.094960 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:13.095286 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 02:29:13.095411 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:29:13.096375 22844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:29:13.098781 22768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } attempt: 3
Jun 19 02:29:13 dist-test-slave-xkrt krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 02:29:13 dist-test-slave-xkrt krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 02:29:13 dist-test-slave-xkrt krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 02:29:13 dist-test-slave-xkrt krb5kdc[22612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 02:29:13.105746 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.095484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:57056)
0619 02:29:13.095630 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:13.095643 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:13.095714 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:13.096804 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:13.096807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:13.096973 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:13.097076 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:13.097083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.098212 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.098217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:13.098953 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:13.098964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.099759 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.099763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:13.099839 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.100273 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:13.100292 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:13.101835 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:13.103950 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:13.103953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:13.103955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:13.104191 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:13.104570 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:13.104575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:13.104577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:13.104672 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:13.105202 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:13.105205 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:13.105293 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 02:29:13.105399 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 02:29:13.105575 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 02:29:13.106468 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.096444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.0.98.60:38111)
0619 02:29:13.096575 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.096579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.096596 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:29:13.096623 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.096625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.096655 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.097175 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.097249 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.098081 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.099101 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.099644 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.100039 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.101995 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:13.102000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:13.102003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:13.102028 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:13.103827 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.104366 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.104370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.104372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.104446 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.104806 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.104810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.104815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.105066 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:13.106027 (+ 961us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:13.106120 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 02:29:13.106281 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 02:29:13.108582 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.096109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.0.98.61:40709)
0619 02:29:13.096246 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.096253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.096266 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:29:13.096298 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.096301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.096346 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.096402 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.097332 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.098084 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.099613 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.099972 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.101798 (+ 1826us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.104044 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:13.104051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:13.104054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:13.104087 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:13.107078 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.107525 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.107528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.107530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.107598 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.107873 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.107875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.107876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.108020 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:13.108189 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:13.108343 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 02:29:13.108432 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:29:13.109679 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1: Bootstrap starting.
I20260619 02:29:13.111539 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.096865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:57062)
0619 02:29:13.097012 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:13.097024 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:13.097086 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:13.097796 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:13.097801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:13.097948 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:13.098007 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:13.098012 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.099571 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.099576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:13.100439 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:13.100448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.101119 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.101124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:13.101242 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.102001 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:13.102015 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:13.104276 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:13.107027 (+ 2751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:13.107030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:13.107033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:13.110076 (+ 3043us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:13.110389 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:13.110393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:13.110395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:13.110486 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:13.110978 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:13.110980 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:13.111075 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 02:29:13.111184 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 02:29:13.111320 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":1,"spinlock_wait_cycles":2816}
I20260619 02:29:13.112164 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.096963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.0.98.60:38111)
0619 02:29:13.097450 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.097457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.097468 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:29:13.097496 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.097498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.097626 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.097737 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.098128 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.099442 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.100567 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.100942 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.101027 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.104393 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:13.104396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:13.104401 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:13.104425 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:13.106904 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.110200 (+ 3296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.110206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.110209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.110288 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.110636 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.110641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.110647 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.110857 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:13.110945 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:13.111724 (+ 779us) server_negotiation.cc:299] Negotiation successful
0619 02:29:13.111830 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":136,"threads_started":1}
I20260619 02:29:13.112480 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.099114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38111 (local address 127.0.0.1:57066)
0619 02:29:13.099329 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:13.099344 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:13.099414 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:13.099792 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:13.099795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:13.099976 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:13.100094 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:13.100100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.101267 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.101270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:13.102218 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:13.102230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.102864 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.102868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:13.102956 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.103421 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:13.103463 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:13.105247 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:13.110409 (+ 5162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:13.110412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:13.110414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:13.110695 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:13.111410 (+ 715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:13.111413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:13.111416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:13.111500 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:13.111944 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:13.111947 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:13.112055 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 02:29:13.112141 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:29:13.112288 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260619 02:29:13.113001 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.099298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57066 (local address 127.0.98.60:38111)
0619 02:29:13.099542 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.099553 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.099566 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:29:13.099595 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.099598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.099641 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.100205 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.100321 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.101142 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.102371 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.102742 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.102862 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.105405 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:13.105409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:13.105411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:13.105432 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:13.110286 (+ 4854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.111210 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.111213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.111215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.111297 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.111625 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.111628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.111630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.111824 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:13.112651 (+ 827us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:13.112730 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 02:29:13.112841 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
I20260619 02:29:13.113126 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:29:13.113718 22693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:13.114660 22693 log.cc:824] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:13.114985 22844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9: Bootstrap starting.
I20260619 02:29:13.116899 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1: No bootstrap required, opened a new log
I20260619 02:29:13.117494 22844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:13.118332 22844 log.cc:824] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:13.120196 22693 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } }
I20260619 02:29:13.120406 22693 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0eb5f5b151b461aa93848ddb5d726c1, State: Initialized, Role: FOLLOWER
I20260619 02:29:13.120584 22768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf: Bootstrap starting.
I20260619 02:29:13.120754 22693 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } }
I20260619 02:29:13.121208 22853 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } } }
I20260619 02:29:13.121342 22853 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:29:13.121701 22667 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a0eb5f5b151b461aa93848ddb5d726c1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46400
I20260619 02:29:13.122027 22693 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:13.122128 22667 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 02:29:13.122246 22667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:29:13.123029 22768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:13.123294 22844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9: No bootstrap required, opened a new log
I20260619 02:29:13.123512 22667 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } }
I20260619 02:29:13.123811 22768 log.cc:824] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:13.124464 22667 leader_election.cc:288] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [CANDIDATE]: Term 1 election: Requested vote from peers 85bbc9b3b2914d55be8e315d1a0cfedf (127.0.98.61:40709), 784538c8aff549d0bcac7937ee08e0c9 (127.0.98.60:38111)
W20260619 02:29:13.124744 22646 tablet.cc:2369] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:29:13.126107 22768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf: No bootstrap required, opened a new log
I20260619 02:29:13.130268 22768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } }
I20260619 02:29:13.130467 22768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85bbc9b3b2914d55be8e315d1a0cfedf, State: Initialized, Role: FOLLOWER
I20260619 02:29:13.130791 22768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } }
I20260619 02:29:13.127832 22625 master_runner.cc:186] Error getting master registration for 127.0.98.61:40709: OK, instance_id { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" instance_seqno: 1781836152849894 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:29:13.129525 22844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } }
I20260619 02:29:13.131304 22625 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:29:13.131337 22844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 784538c8aff549d0bcac7937ee08e0c9, State: Initialized, Role: FOLLOWER
I20260619 02:29:13.131616 22844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } }
I20260619 02:29:13.132069 22768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:13.132129 22742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf"
I20260619 02:29:13.132349 22742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:29:13.132869 22844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:13.133678 22742 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0eb5f5b151b461aa93848ddb5d726c1 in term 1.
I20260619 02:29:13.134030 22815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "784538c8aff549d0bcac7937ee08e0c9"
I20260619 02:29:13.134223 22815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:29:13.135530 22815 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0eb5f5b151b461aa93848ddb5d726c1 in term 1.
I20260619 02:29:13.136160 22642 leader_election.cc:302] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 784538c8aff549d0bcac7937ee08e0c9, a0eb5f5b151b461aa93848ddb5d726c1; no voters:
I20260619 02:29:13.136839 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } } }
I20260619 02:29:13.136950 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:29:13.139132 22890 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:29:13.139207 22890 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 02:29:13.139783 22864 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:29:13.139860 22864 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 02:29:13.142534 22896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:29:13.142607 22896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:29:13.143908 22853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:29:13.144145 22853 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [term 1 LEADER]: Becoming Leader. State: Replica: a0eb5f5b151b461aa93848ddb5d726c1, State: Running, Role: LEADER
I20260619 02:29:13.144466 22853 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } }
I20260619 02:29:13.144667 22776 master_runner.cc:263] No leader master found from master 784538c8aff549d0bcac7937ee08e0c9
I20260619 02:29:13.144730 22776 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:29:13.144838 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.130323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48718 (local address 127.0.98.61:40709)
0619 02:29:13.130428 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.130433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.135674 (+ 5241us) server_negotiation.cc:407] Connection header received
0619 02:29:13.135869 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.135872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.135923 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.135989 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.136939 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.137656 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.138712 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.139038 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.139111 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.140938 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:13.140941 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:13.140944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:13.140968 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:13.142692 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.143155 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.143158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.143161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.143246 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.143593 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.143596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.143597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.143821 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:13.143932 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:13.144400 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 02:29:13.144524 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:29:13.145299 22700 master_runner.cc:263] No leader master found from master 85bbc9b3b2914d55be8e315d1a0cfedf
I20260619 02:29:13.145362 22700 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:29:13.146306 22873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } } }
I20260619 02:29:13.146624 22853 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0eb5f5b151b461aa93848ddb5d726c1. Latest consensus state: current_term: 1 leader_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } } }
I20260619 02:29:13.146761 22853 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:13.146440 22873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:29:13.148231 22902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:29:13.148991 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.129885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46404 (local address 127.0.98.62:36569)
0619 02:29:13.130105 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.130109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.130585 (+ 476us) server_negotiation.cc:407] Connection header received
0619 02:29:13.131262 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.131265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.131394 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.131525 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.132604 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.133349 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.134574 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.134875 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.134949 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.145453 (+ 10504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:13.145457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:13.145460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:13.145485 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:13.147106 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.147673 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.147676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.147678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.147759 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.148071 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.148075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.148076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.148292 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:13.148402 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:13.148731 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 02:29:13.148848 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:29:13.150333 22902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:29:13.150485 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:13.131004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57074 (local address 127.0.98.60:38111)
0619 02:29:13.131109 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.131113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.139741 (+ 8628us) server_negotiation.cc:407] Connection header received
0619 02:29:13.139901 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.139904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.139949 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.140015 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.140828 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.141566 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.142617 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:13.142973 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:13.143052 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:13.146954 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:13.146958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:13.146963 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:13.146987 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:13.148656 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.149124 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.149127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.149129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.149213 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:13.149512 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:13.149515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:13.149517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:13.149763 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:13.149867 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:13.150226 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 02:29:13.150340 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:29:13.155640 22815 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9 [term 1 FOLLOWER]: Refusing update from remote peer a0eb5f5b151b461aa93848ddb5d726c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:29:13.155711 22742 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf [term 1 FOLLOWER]: Refusing update from remote peer a0eb5f5b151b461aa93848ddb5d726c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:29:13.157836 22900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260619 02:29:13.158013 22853 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:29:13.161942 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0eb5f5b151b461aa93848ddb5d726c1. Latest consensus state: current_term: 1 leader_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } } }
I20260619 02:29:13.162063 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:29:13.162026 22873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0eb5f5b151b461aa93848ddb5d726c1. Latest consensus state: current_term: 1 leader_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } } }
I20260619 02:29:13.162130 22873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:29:13.165616 22853 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } } }
I20260619 02:29:13.165733 22853 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:13.165936 22903 mvcc.cc:205] Tried to move back new op lower bound from 7298400883319431168 to 7298400883283857408. Current Snapshot: MvccSnapshot[applied={T|T < 7298400883319431168}]
I20260619 02:29:13.166707 22873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } } }
I20260619 02:29:13.166812 22873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:29:13.168740 22905 mvcc.cc:205] Tried to move back new op lower bound from 7298400883319431168 to 7298400883283857408. Current Snapshot: MvccSnapshot[applied={T|T < 7298400883319431168}]
I20260619 02:29:13.172966 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } } }
I20260619 02:29:13.173079 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1 [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:13.174124 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0eb5f5b151b461aa93848ddb5d726c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36569 } } peers { permanent_uuid: "85bbc9b3b2914d55be8e315d1a0cfedf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "784538c8aff549d0bcac7937ee08e0c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38111 } } }
I20260619 02:29:13.174237 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:29:13.174849 22904 mvcc.cc:205] Tried to move back new op lower bound from 7298400883319431168 to 7298400883283857408. Current Snapshot: MvccSnapshot[applied={T|T < 7298400883319431168}]
I20260619 02:29:13.179375 22902 catalog_manager.cc:1347] Generated new cluster ID: 87c25dc5be404d0cb1da9c26084de15a
I20260619 02:29:13.179435 22902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:29:13.194757 22902 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:29:13.198325 22902 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:29:13.225629 22902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0eb5f5b151b461aa93848ddb5d726c1: Generated new TSK 0
I20260619 02:29:13.228837 22902 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:29:13.262863 22695 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:13.259515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.0.98.62:36569)
0619 02:29:13.259643 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.259648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.261480 (+ 1832us) server_negotiation.cc:407] Connection header received
0619 02:29:13.261967 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.261971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.262025 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.262082 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.262722 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46414: BlockingRecv error: recv got EOF from 127.0.0.1:46414 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 02:29:13.264132 22774 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:13.260373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48724 (local address 127.0.98.61:40709)
0619 02:29:13.260542 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.260547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.261869 (+ 1322us) server_negotiation.cc:407] Connection header received
0619 02:29:13.263406 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.263409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.263476 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.263524 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.264014 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48724: BlockingRecv error: recv got EOF from 127.0.0.1:48724 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 02:29:13.265806 22850 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:13.261187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.0.98.60:38111)
0619 02:29:13.261313 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.261317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.264798 (+ 3481us) server_negotiation.cc:407] Connection header received
0619 02:29:13.265009 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.265013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.265060 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.265110 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.265691 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57082: BlockingRecv error: recv got EOF from 127.0.0.1:57082 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 02:29:13.269734 22774 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:13.268243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48736 (local address 127.0.98.61:40709)
0619 02:29:13.268339 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.268343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.268827 (+ 484us) server_negotiation.cc:407] Connection header received
0619 02:29:13.269050 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.269053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.269122 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.269173 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.269641 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48736: BlockingRecv error: recv got EOF from 127.0.0.1:48736 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 02:29:13.274518 22774 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:13.273092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.0.98.61:40709)
0619 02:29:13.273185 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.273189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.273710 (+ 521us) server_negotiation.cc:407] Connection header received
0619 02:29:13.273881 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.273884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.273940 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.273989 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.274427 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48752: BlockingRecv error: recv got EOF from 127.0.0.1:48752 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 02:29:13.279727 22774 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:13.278382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48768 (local address 127.0.98.61:40709)
0619 02:29:13.278472 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.278476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.278953 (+ 477us) server_negotiation.cc:407] Connection header received
0619 02:29:13.279120 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.279123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.279164 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.279210 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.279641 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48768: BlockingRecv error: recv got EOF from 127.0.0.1:48768 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 02:29:13.287209 22774 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:13.285706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48778 (local address 127.0.98.61:40709)
0619 02:29:13.285829 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.285834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.286344 (+ 510us) server_negotiation.cc:407] Connection header received
0619 02:29:13.286528 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.286533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.286580 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.286631 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.287115 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48778: BlockingRecv error: recv got EOF from 127.0.0.1:48778 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 02:29:13.299168 22774 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:13.297389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.0.98.61:40709)
0619 02:29:13.297502 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.297507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.298123 (+ 616us) server_negotiation.cc:407] Connection header received
0619 02:29:13.298415 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.298419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.298473 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.298523 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.299054 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48784: BlockingRecv error: recv got EOF from 127.0.0.1:48784 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 02:29:13.319988 22774 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:13.317845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.0.98.61:40709)
0619 02:29:13.317983 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.317987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.318867 (+ 880us) server_negotiation.cc:407] Connection header received
0619 02:29:13.319166 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.319170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.319226 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.319285 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.319854 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48792: BlockingRecv error: recv got EOF from 127.0.0.1:48792 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 02:29:13.355786 22774 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:13.354470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48796 (local address 127.0.98.61:40709)
0619 02:29:13.354608 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.354614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.354867 (+ 253us) server_negotiation.cc:407] Connection header received
0619 02:29:13.355024 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.355028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.355084 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.355178 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.355664 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48796: BlockingRecv error: recv got EOF from 127.0.0.1:48796 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 02:29:13.423285 22774 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:13.422097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48798 (local address 127.0.98.61:40709)
0619 02:29:13.422232 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.422237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.422443 (+ 206us) server_negotiation.cc:407] Connection header received
0619 02:29:13.422542 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.422545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.422591 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.422698 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.423136 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48798: BlockingRecv error: recv got EOF from 127.0.0.1:48798 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 02:29:13.554399 22774 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:13.553313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.0.98.61:40709)
0619 02:29:13.553457 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.553460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.553584 (+ 124us) server_negotiation.cc:407] Connection header received
0619 02:29:13.553667 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.553670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.553721 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.553802 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.554245 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48800: BlockingRecv error: recv got EOF from 127.0.0.1:48800 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 02:29:13.813506 22774 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:13.812518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.0.98.61:40709)
0619 02:29:13.812659 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:13.812663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:13.812678 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:29:13.812813 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:13.812815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:13.812861 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:13.812932 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:13.813360 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48810: BlockingRecv error: recv got EOF from 127.0.0.1:48810 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:29:14.140618 22890 catalog_manager.cc:1259] Loaded cluster ID: 87c25dc5be404d0cb1da9c26084de15a
I20260619 02:29:14.140714 22890 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf: loading cluster ID for follower catalog manager: success
I20260619 02:29:14.142421 22890 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf: acquiring CA information for follower catalog manager: success
I20260619 02:29:14.143922 22896 catalog_manager.cc:1259] Loaded cluster ID: 87c25dc5be404d0cb1da9c26084de15a
I20260619 02:29:14.143985 22896 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9: loading cluster ID for follower catalog manager: success
I20260619 02:29:14.143989 22890 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85bbc9b3b2914d55be8e315d1a0cfedf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:29:14.145630 22896 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9: acquiring CA information for follower catalog manager: success
I20260619 02:29:14.146587 22896 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 784538c8aff549d0bcac7937ee08e0c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:29:14.331156 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:14.327401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.0.98.61:40709)
0619 02:29:14.327660 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:14.327664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:14.327680 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:29:14.327816 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:14.327819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:14.327944 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:14.328055 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:29:14.328307 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:14.328905 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:14.329746 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:14.329990 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:14.330075 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:14.330325 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:29:14.330748 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:29:14.330839 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:14.330890 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 02:29:14.330937 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
I20260619 02:29:14.338209 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:14.334529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.0.98.60:38111)
0619 02:29:14.334779 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:14.334783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:14.334797 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:29:14.334967 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:14.334970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:14.335088 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:14.335199 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:29:14.335446 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:14.335946 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:14.336767 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:14.336980 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:14.337053 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:14.337317 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:29:14.337833 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:29:14.337925 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:14.338030 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 02:29:14.338078 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
I20260619 02:29:14.344797 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:14.341361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48828 (local address 127.0.98.61:40709)
0619 02:29:14.341473 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:14.341478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:14.341633 (+ 155us) server_negotiation.cc:407] Connection header received
0619 02:29:14.341816 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:14.341820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:14.341882 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:14.341956 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:29:14.342185 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:14.342723 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:14.343683 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:14.343904 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:14.344026 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:14.344312 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:29:14.344396 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:29:14.344472 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:14.344586 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 02:29:14.344637 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:29:14.345330 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:14.341191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.0.98.60:38111)
0619 02:29:14.341311 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:14.341318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:14.341603 (+ 285us) server_negotiation.cc:407] Connection header received
0619 02:29:14.341717 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:14.341723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:14.341780 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:14.341845 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:29:14.342066 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:14.342842 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:14.344021 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:14.344343 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:14.344406 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:14.344743 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:29:14.344811 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:29:14.344857 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:14.345135 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 02:29:14.345184 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:29:14.346273 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:14.341000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.0.98.62:36569)
0619 02:29:14.341485 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:14.341490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:14.341504 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:29:14.341627 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:14.341630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:14.341816 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:14.341934 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:29:14.342324 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:14.343080 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:14.344287 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:14.344682 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:14.344770 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:14.345191 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:29:14.345777 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:29:14.345868 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:14.346114 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 02:29:14.346166 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":61,"threads_started":1}
I20260619 02:29:14.349203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 22625
I20260619 02:29:14.355333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 22700
I20260619 02:29:14.361290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 22776
2026-06-19T02:29:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3065 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:29:14 dist-test-slave-xkrt krb5kdc[22993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:29:14 dist-test-slave-xkrt krb5kdc[22993](info): set up 2 sockets
Jun 19 02:29:14 dist-test-slave-xkrt krb5kdc[22993](info): commencing operation
krb5kdc: starting...
W20260619 02:29:15.390110 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/test-user.keytab.
Jun 19 02:29:15 dist-test-slave-xkrt krb5kdc[22993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:29:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 02:29:15.516592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33049
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33049,127.0.98.61:39213,127.0.98.60:36839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:29:15.625643 23006 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:29:15.625999 23006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:29:15.626070 23006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:29:15.626152 23006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:29:15.630121 23006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:29:15.630208 23006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:29:15.630247 23006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:29:15.630283 23006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:29:15.630314 23006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:29:15.635591 23006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33049,127.0.98.61:39213,127.0.98.60:36839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:29:15.636714 23006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:29:15.637766 23006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:15.646265 23011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:15.646255 23012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:15.646371 23006 server_base.cc:1034] running on GCE node
W20260619 02:29:15.646256 23014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:15.647061 23006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:15.648252 23006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:15.649456 23006 hybrid_clock.cc:648] HybridClock initialized: now 1781836155649442 us; error 26 us; skew 500 ppm
Jun 19 02:29:15 dist-test-slave-xkrt krb5kdc[22993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:15.652376 23006 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 02:29:15.653322 23006 webserver.cc:460] Webserver started at http://127.0.98.62:43189/ using document root <none> and password file <none>
I20260619 02:29:15.653956 23006 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:15.654047 23006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:15.654284 23006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:15.656292 23006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/data/instance:
uuid: "1a15e4bacead4329b6c7369517b0fb27"
format_stamp: "Formatted at 2026-06-19 02:29:15 on dist-test-slave-xkrt"
server_key: "1585bac44fedb52acd3ab0b599e214b8"
server_key_iv: "8464fd4a189f235d4e0e8efb3178a4c1"
server_key_version: "encryptionkey@0"
I20260619 02:29:15.656855 23006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a15e4bacead4329b6c7369517b0fb27"
format_stamp: "Formatted at 2026-06-19 02:29:15 on dist-test-slave-xkrt"
server_key: "1585bac44fedb52acd3ab0b599e214b8"
server_key_iv: "8464fd4a189f235d4e0e8efb3178a4c1"
server_key_version: "encryptionkey@0"
I20260619 02:29:15.660465 23006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 02:29:15.662823 23021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:15.664012 23006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:29:15.664155 23006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/wal
uuid: "1a15e4bacead4329b6c7369517b0fb27"
format_stamp: "Formatted at 2026-06-19 02:29:15 on dist-test-slave-xkrt"
server_key: "1585bac44fedb52acd3ab0b599e214b8"
server_key_iv: "8464fd4a189f235d4e0e8efb3178a4c1"
server_key_version: "encryptionkey@0"
I20260619 02:29:15.664263 23006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:15.682379 23006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:15.685765 23006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:15.685978 23006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:15.692974 23006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33049
I20260619 02:29:15.693274 23073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33049 every 8 connection(s)
I20260619 02:29:15.693985 23006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/data/info.pb
I20260619 02:29:15.696779 23074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:15.700405 23074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:29:15.703172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 23006
I20260619 02:29:15.703332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-0/wal/instance
I20260619 02:29:15.703652 392 external_mini_cluster.cc:1428] Setting key 3faf90ee65c79f00e7109a9fb3c83e92
Jun 19 02:29:15 dist-test-slave-xkrt krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260619 02:29:15.717178 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.702618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.0.98.62:33049)
0619 02:29:15.703158 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:15.703177 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:15.703358 (+ 181us) server_negotiation.cc:407] Connection header received
0619 02:29:15.704420 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:15.704428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:15.704682 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:15.704887 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:15.705952 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.706918 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.708409 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.709158 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.709381 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:15.712063 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:15.712072 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:15.712089 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:15.712141 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:15.714243 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:15.714691 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:15.714700 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:15.714705 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:15.714850 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:15.715146 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:15.715149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:15.715151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:15.715682 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:15.715826 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:15.716197 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 02:29:15.716424 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":367,"thread_start_us":138,"threads_started":1}
I20260619 02:29:15.717177 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.702612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33049 (local address 127.0.0.1:59018)
0619 02:29:15.703205 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.703252 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:15.704112 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:15.704896 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:15.704900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:15.705363 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:15.705763 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:15.705784 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.707060 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.707066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:15.708278 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:15.708287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.709318 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.709321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:15.709409 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:15.710146 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:15.710161 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:15.711892 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:15.714355 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:15.714362 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:15.714370 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:15.714579 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:15.714953 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:15.714955 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:15.714956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:15.715052 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:15.715818 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:15.715823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:15.715939 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 02:29:15.716073 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 02:29:15.716330 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"spinlock_wait_cycles":73088,"thread_start_us":158,"threads_started":1}
I20260619 02:29:15.719669 23074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:29:15.720470 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.720149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39213 (local address 127.0.0.1:53430)
0619 02:29:15.720293 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.720386 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 02:29:15.720682 23022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39213: connect: Connection refused (error 111)
I20260619 02:29:15.721731 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.721437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39213 (local address 127.0.0.1:53444)
0619 02:29:15.721609 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.721665 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 02:29:15.721904 23074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39213: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39213: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 02:29:15.730551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39213
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33049,127.0.98.61:39213,127.0.98.60:36839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:29:15.771184 23074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } attempt: 1
I20260619 02:29:15.772177 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.771803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39213 (local address 127.0.0.1:53446)
0619 02:29:15.771997 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.772070 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 02:29:15.772907 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.772681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39213 (local address 127.0.0.1:53458)
0619 02:29:15.772789 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.772841 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 02:29:15.773038 23074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39213: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39213: connect: Connection refused (error 111)
W20260619 02:29:15.840862 23081 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:29:15.841140 23081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:29:15.841203 23081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:29:15.841264 23081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 02:29:15.841212 23074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } attempt: 2
I20260619 02:29:15.842015 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.841711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39213 (local address 127.0.0.1:53466)
0619 02:29:15.841868 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.841933 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 02:29:15.843192 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.843007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39213 (local address 127.0.0.1:53474)
0619 02:29:15.843098 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.843142 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 02:29:15.843329 23074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39213: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39213: connect: Connection refused (error 111)
W20260619 02:29:15.845957 23081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:29:15.846005 23081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:29:15.846025 23081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:29:15.846079 23081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:29:15.846138 23081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:29:15.851326 23081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33049,127.0.98.61:39213,127.0.98.60:36839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:29:15.852344 23081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:29:15.853332 23081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:15.861727 23086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:15.861709 23087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:15.861709 23089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:15.862098 23081 server_base.cc:1034] running on GCE node
I20260619 02:29:15.862653 23081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:15.863762 23081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:15.864967 23081 hybrid_clock.cc:648] HybridClock initialized: now 1781836155864938 us; error 49 us; skew 500 ppm
Jun 19 02:29:15 dist-test-slave-xkrt krb5kdc[22993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:15.867774 23081 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 02:29:15.868709 23081 webserver.cc:460] Webserver started at http://127.0.98.61:37053/ using document root <none> and password file <none>
I20260619 02:29:15.869213 23081 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:15.869271 23081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:15.869434 23081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:15.871471 23081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/data/instance:
uuid: "4da9e9ffabe64406ac8ceccdafab4e6a"
format_stamp: "Formatted at 2026-06-19 02:29:15 on dist-test-slave-xkrt"
server_key: "1eb9f3d22eb7ada22ed1607f35e05ce0"
server_key_iv: "c97474c017ae15e8bccca951a1d31a5f"
server_key_version: "encryptionkey@0"
I20260619 02:29:15.871964 23081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/wal/instance:
uuid: "4da9e9ffabe64406ac8ceccdafab4e6a"
format_stamp: "Formatted at 2026-06-19 02:29:15 on dist-test-slave-xkrt"
server_key: "1eb9f3d22eb7ada22ed1607f35e05ce0"
server_key_iv: "c97474c017ae15e8bccca951a1d31a5f"
server_key_version: "encryptionkey@0"
I20260619 02:29:15.875600 23081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:29:15.878086 23096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:15.879364 23081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:29:15.879464 23081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/wal
uuid: "4da9e9ffabe64406ac8ceccdafab4e6a"
format_stamp: "Formatted at 2026-06-19 02:29:15 on dist-test-slave-xkrt"
server_key: "1eb9f3d22eb7ada22ed1607f35e05ce0"
server_key_iv: "c97474c017ae15e8bccca951a1d31a5f"
server_key_version: "encryptionkey@0"
I20260619 02:29:15.879588 23081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:15.906740 23081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:15.909703 23081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:15.909859 23081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:15.916736 23081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39213
I20260619 02:29:15.917023 23148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39213 every 8 connection(s)
I20260619 02:29:15.917783 23081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/data/info.pb
I20260619 02:29:15.920449 23149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:15.924072 23149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:29:15.926640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 23081
I20260619 02:29:15.926785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-1/wal/instance
I20260619 02:29:15.927066 392 external_mini_cluster.cc:1428] Setting key 3493d9f8049d878804fb4a551fca76ca
Jun 19 02:29:15 dist-test-slave-xkrt krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 02:29:15.934526 23074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260619 02:29:15.939411 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.925950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.0.98.62:33049)
0619 02:29:15.926088 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:15.926096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:15.927015 (+ 919us) server_negotiation.cc:407] Connection header received
0619 02:29:15.928091 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:15.928094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:15.928163 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:15.928285 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:15.929491 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.930161 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.931388 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.931673 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.931796 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:15.934566 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:15.934569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:15.934571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:15.934597 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:15.936695 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:15.937441 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:15.937448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:15.937453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:15.937579 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:15.938041 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:15.938044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:15.938046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:15.938283 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:15.938381 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:15.939048 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 02:29:15.939204 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 02:29:15 dist-test-slave-xkrt krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 02:29:15.941108 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.926239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33049 (local address 127.0.0.1:59024)
0619 02:29:15.926816 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.926879 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:15.927849 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:15.928467 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:15.928474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:15.928838 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:15.929330 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:15.929343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.930308 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.930311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:15.931252 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:15.931262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.931837 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.931841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:15.932030 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:15.932674 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:15.932691 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:15.934362 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:15.936830 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:15.936836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:15.936850 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:15.937303 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:15.937712 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:15.937719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:15.937725 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:15.937932 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:15.938395 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:15.938404 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:15.938744 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 02:29:15.939631 (+ 887us) client_negotiation.cc:240] Negotiation successful
0619 02:29:15.939979 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":167,"threads_started":1}
I20260619 02:29:15.943193 23149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:29:15.944819 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.935003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39213 (local address 127.0.0.1:53476)
0619 02:29:15.935152 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.935169 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:15.935255 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:15.935941 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:15.935945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:15.936169 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:15.936267 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:15.936272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.937479 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.937484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:15.938125 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:15.938134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.938865 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.938869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:15.938990 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:15.939494 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:15.939515 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:15.940855 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:15.943141 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:15.943146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:15.943150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:15.943411 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:15.943744 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:15.943749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:15.943751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:15.943847 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:15.944423 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:15.944426 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:15.944527 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 02:29:15.944608 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:29:15.944709 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 02:29:15.945533 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.935289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.0.98.61:39213)
0619 02:29:15.935605 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:15.935614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:15.935634 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:29:15.935668 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:15.935672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:15.935813 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:15.935959 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:15.936377 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.937351 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.938246 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.938739 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.938868 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:15.941032 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:15.941040 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:15.941050 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:15.941072 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:15.943018 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:15.943540 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:15.943547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:15.943550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:15.943640 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:15.943972 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:15.943975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:15.943977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:15.944277 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:15.945073 (+ 796us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:15.945180 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 02:29:15.945328 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":136,"threads_started":1}
I20260619 02:29:15.947492 23074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:29:15.948268 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.947967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48402)
0619 02:29:15.948116 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.948181 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 02:29:15.949051 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.948750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48408)
0619 02:29:15.948948 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.948990 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260619 02:29:15.949251 23074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36839: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
Jun 19 02:29:15 dist-test-slave-xkrt krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 02:29:15.953099 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.943775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39213 (local address 127.0.0.1:53480)
0619 02:29:15.943893 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.943920 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:15.943988 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:15.944449 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:15.944454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:15.944637 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:15.944765 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:15.944775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.946090 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.946094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:15.946712 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:15.946720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.947971 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.947975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:15.948077 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:15.948562 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:15.948572 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:15.949997 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:15.951661 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:15.951663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:15.951665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:15.951881 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:15.952178 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:15.952179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:15.952181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:15.952248 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:15.952646 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:15.952648 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:15.952726 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 02:29:15.952869 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 02:29:15.952995 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 02:29:15.953236 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.943975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.0.98.61:39213)
0619 02:29:15.944212 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:15.944220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:15.944233 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:29:15.944266 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:15.944269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:15.944310 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:15.944406 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:15.945144 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.945980 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.946827 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:15.947841 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:15.947952 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:15.950131 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:15.950134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:15.950136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:15.950157 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:15.951555 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:15.951986 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:15.951989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:15.951990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:15.952090 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:15.952352 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:15.952355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:15.952356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:15.952533 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:15.952614 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:15.952956 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 02:29:15.953053 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":85,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 02:29:15.954113 23149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:29:15.954867 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.954583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48422)
0619 02:29:15.954698 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.954787 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 02:29:15.955049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
/tmp/dist-test-taskjWwzrN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36839
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33049,127.0.98.61:39213,127.0.98.60:36839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:29:15.955096 23098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
I20260619 02:29:15.956054 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.955761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48424)
0619 02:29:15.955915 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.955977 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:29:15.956207 23149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36839: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
I20260619 02:29:15.980429 23074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } attempt: 1
I20260619 02:29:15.981303 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.980988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48428)
0619 02:29:15.981157 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.981219 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 02:29:15.982234 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:15.981970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48432)
0619 02:29:15.982080 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:15.982152 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 02:29:15.982359 23074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36839: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
I20260619 02:29:16.005527 23149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } attempt: 1
I20260619 02:29:16.006510 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.006210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48446)
0619 02:29:16.006367 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:16.006431 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 02:29:16.007185 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.007003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48460)
0619 02:29:16.007097 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:16.007138 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 02:29:16.007341 23149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36839: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
I20260619 02:29:16.057533 23074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } attempt: 2
I20260619 02:29:16.058427 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.058132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48464)
0619 02:29:16.058283 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:16.058349 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 02:29:16.059180 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.058991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48478)
0619 02:29:16.059089 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:16.059132 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 02:29:16.059321 23074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36839: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
W20260619 02:29:16.066207 23157 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:29:16.066478 23157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:29:16.066530 23157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:29:16.066622 23157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:29:16.070880 23157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:29:16.070932 23157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:29:16.070951 23157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:29:16.070992 23157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:29:16.071024 23157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:29:16.075590 23149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } attempt: 2
I20260619 02:29:16.076412 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.076118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48480)
0619 02:29:16.076268 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:16.076331 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 02:29:16.076658 23157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33049,127.0.98.61:39213,127.0.98.60:36839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 01:50:15 UTC on ccycloud.quasar-qviwlx.root.comops.site
build id 4454979
I20260619 02:29:16.077275 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.077051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48496)
0619 02:29:16.077173 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:16.077225 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 02:29:16.077443 23149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36839: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36839: connect: Connection refused (error 111)
I20260619 02:29:16.077862 23157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:29:16.078873 23157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:29:16.087467 23165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:16.087478 23163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:29:16.087467 23162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:29:16.087711 23157 server_base.cc:1034] running on GCE node
I20260619 02:29:16.088383 23157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:29:16.089474 23157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:29:16.090636 23157 hybrid_clock.cc:648] HybridClock initialized: now 1781836156090620 us; error 30 us; skew 500 ppm
Jun 19 02:29:16 dist-test-slave-xkrt krb5kdc[22993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:29:16.093477 23157 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 02:29:16.094396 23157 webserver.cc:460] Webserver started at http://127.0.98.60:43643/ using document root <none> and password file <none>
I20260619 02:29:16.094966 23157 fs_manager.cc:359] Metadata directory not provided
I20260619 02:29:16.095029 23157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:29:16.095242 23157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:29:16.097218 23157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/data/instance:
uuid: "2f9e1f9156be4c638256be0f5f20cd46"
format_stamp: "Formatted at 2026-06-19 02:29:16 on dist-test-slave-xkrt"
server_key: "341b0913e78fdb320cc5385d65f84974"
server_key_iv: "175a330f339a06636d7561fd87e530ff"
server_key_version: "encryptionkey@0"
I20260619 02:29:16.097788 23157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/wal/instance:
uuid: "2f9e1f9156be4c638256be0f5f20cd46"
format_stamp: "Formatted at 2026-06-19 02:29:16 on dist-test-slave-xkrt"
server_key: "341b0913e78fdb320cc5385d65f84974"
server_key_iv: "175a330f339a06636d7561fd87e530ff"
server_key_version: "encryptionkey@0"
I20260619 02:29:16.101259 23157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:29:16.103574 23172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:29:16.104732 23157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:29:16.104859 23157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/wal
uuid: "2f9e1f9156be4c638256be0f5f20cd46"
format_stamp: "Formatted at 2026-06-19 02:29:16 on dist-test-slave-xkrt"
server_key: "341b0913e78fdb320cc5385d65f84974"
server_key_iv: "175a330f339a06636d7561fd87e530ff"
server_key_version: "encryptionkey@0"
I20260619 02:29:16.104974 23157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:29:16.120759 23157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:29:16.123955 23157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:29:16.124112 23157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:29:16.130908 23157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36839
I20260619 02:29:16.131193 23224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36839 every 8 connection(s)
I20260619 02:29:16.131891 23157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/data/info.pb
I20260619 02:29:16.134673 23225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:29:16.138234 23225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:29:16.141322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu as pid 23157
I20260619 02:29:16.141469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjWwzrN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781835873883564-392-0/minicluster-data/master-2/wal/instance
I20260619 02:29:16.141765 392 external_mini_cluster.cc:1428] Setting key 1e312339cda5f11826ef12774fd2635e
Jun 19 02:29:16 dist-test-slave-xkrt krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 02:29:16 dist-test-slave-xkrt krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 02:29:16.151718 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.140008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.0.98.62:33049)
0619 02:29:16.140139 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.140143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.140898 (+ 755us) server_negotiation.cc:407] Connection header received
0619 02:29:16.141874 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.141878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.141934 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.142036 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.143442 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.144006 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.145015 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.145222 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.145299 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.147975 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:16.147978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:16.147980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:16.148003 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:16.149611 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.150129 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.150132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.150134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.150219 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.150570 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.150573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.150575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.150788 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:16.150899 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:16.151407 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 02:29:16.151539 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:29:16.152675 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.140342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33049 (local address 127.0.0.1:59028)
0619 02:29:16.140756 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:16.140795 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:16.141675 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:16.142213 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:16.142223 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:16.142702 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:16.143284 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:16.143303 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.144121 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.144123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:16.144899 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:16.144908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.145363 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.145367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:16.145584 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.146198 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:16.146213 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:16.147833 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:16.149702 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:16.149713 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:16.149725 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:16.150031 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:16.150342 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:16.150345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:16.150346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:16.150470 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:16.150893 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:16.150898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:16.151123 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 02:29:16.151347 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 02:29:16.151649 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"spinlock_wait_cycles":768,"thread_start_us":124,"threads_started":1}
I20260619 02:29:16.152851 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.142401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.0.98.60:36839)
0619 02:29:16.142713 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.142723 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.142742 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:29:16.142857 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.142861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.143016 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.143141 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.143929 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.144795 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.145857 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.146330 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.146410 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.149264 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:16.149272 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:16.149286 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:16.149312 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:16.151108 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.151558 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.151563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.151565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.151641 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.151933 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.151936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.151937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.152193 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:16.152390 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:16.152551 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 02:29:16.152680 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"spinlock_wait_cycles":2944,"thread_start_us":132,"threads_started":1}
I20260619 02:29:16.153894 23225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:29:16 dist-test-slave-xkrt krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 02:29:16.158494 23074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } attempt: 3
Jun 19 02:29:16 dist-test-slave-xkrt krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 02:29:16.162837 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.154410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39213 (local address 127.0.0.1:53492)
0619 02:29:16.154622 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:16.154633 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:16.154688 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:16.154922 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:16.154925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:16.155089 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:16.155202 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:16.155209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.155915 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.155922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:16.156428 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:16.156435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.156916 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.156919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:16.157019 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.157485 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:16.157498 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:16.158858 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:16.161156 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:16.161158 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:16.161160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:16.161348 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:16.161688 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:16.161691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:16.161693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:16.161776 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:16.162254 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:16.162257 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:16.162368 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 02:29:16.162512 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 02:29:16.162647 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260619 02:29:16.163098 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.154503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53492 (local address 127.0.98.61:39213)
0619 02:29:16.154678 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.154683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.154699 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:29:16.154753 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.154757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.154808 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.154891 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.155300 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.155809 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.156547 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.156809 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.156918 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.159006 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:16.159011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:16.159014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:16.159039 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:16.161050 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.161445 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.161449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.161451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.161528 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.161917 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.161921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.161924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.162159 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:16.162265 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:16.162823 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 02:29:16.162948 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 02:29:16 dist-test-slave-xkrt krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 02:29:16.163995 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.154956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.0.98.62:33049)
0619 02:29:16.155059 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.155062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.155072 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:29:16.155108 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.155110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.155139 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.155195 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.155779 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.156253 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.157009 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.157333 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.157432 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.159867 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:16.159871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:16.159874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:16.159896 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:16.161616 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.162074 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.162077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.162079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.162168 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.162469 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.162473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.162475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.162668 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:16.162859 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:16.163745 (+ 886us) server_negotiation.cc:299] Negotiation successful
0619 02:29:16.163852 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 02:29:16.164137 23225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:29:16.168432 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.158891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48522)
0619 02:29:16.159049 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:16.159063 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:16.159141 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:16.159469 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:16.159472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:16.159632 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:16.159698 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:16.159703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.160609 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.160615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:16.161737 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:16.161744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.162294 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.162298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:16.162383 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.162801 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:16.162814 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:16.164330 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:16.166210 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:16.166215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:16.166217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:16.166405 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:16.167424 (+ 1019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:16.167427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:16.167429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:16.167520 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:16.167953 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:16.167964 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:16.168052 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 02:29:16.168171 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 02:29:16.168289 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 02:29:16.168632 23149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } attempt: 3
I20260619 02:29:16.169780 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.158974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48522 (local address 127.0.98.60:36839)
0619 02:29:16.159291 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.159295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.159309 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:29:16.159342 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.159345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.159386 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.159454 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.159805 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.160494 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.161835 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.162177 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.162582 (+ 405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.164456 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:16.164461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:16.164463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:16.164498 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:16.166104 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.167246 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.167249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.167251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.167327 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.167649 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.167655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.167659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.167864 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:16.167949 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:16.169490 (+ 1541us) server_negotiation.cc:299] Negotiation successful
0619 02:29:16.169621 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
Jun 19 02:29:16 dist-test-slave-xkrt krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 02:29:16 dist-test-slave-xkrt krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 02:29:16.173278 23074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27: Bootstrap starting.
I20260619 02:29:16.173892 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.164634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.0.98.60:36839)
0619 02:29:16.164942 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.164950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.164960 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:29:16.164993 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.164995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.165023 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.165117 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.165488 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.166721 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.167722 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.168105 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.168194 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.170420 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:16.170426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:16.170431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:16.170459 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:16.172148 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.172594 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.172596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.172598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.172667 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.172901 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.172903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.172905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.173124 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:16.173211 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:16.173635 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 02:29:16.173727 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":113,"threads_started":1}
I20260619 02:29:16.174124 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.164604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48532)
0619 02:29:16.164755 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:16.164769 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:16.164838 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:16.165127 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:16.165130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:16.165295 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:16.165381 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:16.165387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.166817 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.166820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:16.167582 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:16.167592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.168278 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.168281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:16.168329 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.168777 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:16.168790 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:16.170281 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:16.172257 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:16.172260 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:16.172262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:16.172488 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:16.172750 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:16.172752 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:16.172754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:16.172822 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:16.173242 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:16.173245 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:16.173320 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 02:29:16.173418 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 02:29:16.173531 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 02:29:16.174953 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.165437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.0.98.61:39213)
0619 02:29:16.165570 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.165576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.165591 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:29:16.165627 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.165632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.165682 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.165765 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.166510 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.167130 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.168119 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.168436 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.168844 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.171660 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:16.171665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:16.171668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:16.171692 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:16.173171 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.173674 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.173678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.173680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.173761 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.174081 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.174084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.174086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.174309 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:16.174496 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:16.174692 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 02:29:16.174803 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:29:16.176275 23074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:16.177009 23225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46: Bootstrap starting.
I20260619 02:29:16.177171 23074 log.cc:824] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:16.178200 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:29:16.179679 23074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27: No bootstrap required, opened a new log
I20260619 02:29:16.179756 23225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:16.180568 23225 log.cc:824] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:16.179363 23048 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1a15e4bacead4329b6c7369517b0fb27"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59040
Jun 19 02:29:16 dist-test-slave-xkrt krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781836155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 02:29:16.182874 23225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46: No bootstrap required, opened a new log
I20260619 02:29:16.183032 23074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } }
I20260619 02:29:16.183271 23074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a15e4bacead4329b6c7369517b0fb27, State: Initialized, Role: FOLLOWER
I20260619 02:29:16.183607 23074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } }
I20260619 02:29:16.184253 23234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } } }
I20260619 02:29:16.184551 23234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:29:16.184908 23074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:16.185014 23048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 02:29:16.185108 23048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:29:16.186291 23048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } }
W20260619 02:29:16.187912 23027 tablet.cc:2369] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:29:16.188673 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.168970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36839 (local address 127.0.0.1:48546)
0619 02:29:16.169075 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 02:29:16.169089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:29:16.169157 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:29:16.169716 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:29:16.169719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:29:16.169892 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:29:16.169988 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:16.169998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.178346 (+ 8348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.178349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:16.179167 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:29:16.179178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.179861 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.179865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:29:16.179956 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.180470 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:29:16.180488 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:29:16.182022 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:29:16.184391 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:16.184395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:16.184398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:16.184666 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:16.187521 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:29:16.187525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:29:16.187527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:29:16.187627 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:29:16.188155 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:29:16.188158 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:29:16.188257 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 02:29:16.188370 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 02:29:16.188518 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 02:29:16.186920 23225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } }
I20260619 02:29:16.188776 23225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f9e1f9156be4c638256be0f5f20cd46, State: Initialized, Role: FOLLOWER
I20260619 02:29:16.187224 23048 leader_election.cc:288] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [CANDIDATE]: Term 1 election: Requested vote from peers 4da9e9ffabe64406ac8ceccdafab4e6a (127.0.98.61:39213), 2f9e1f9156be4c638256be0f5f20cd46 (127.0.98.60:36839)
I20260619 02:29:16.189492 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.169196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.0.98.60:36839)
0619 02:29:16.169453 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.169470 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.169481 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:29:16.169509 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.169512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.169586 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.170292 (+ 706us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.170408 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.178221 (+ 7813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.179311 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.179731 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.180184 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.182212 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:16.182216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:16.182219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:16.182242 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:16.184264 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.184793 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.184799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.184803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.184891 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.187742 (+ 2851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.187746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.187748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.187977 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:16.188073 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:16.189209 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0619 02:29:16.189333 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":116,"threads_started":1}
W20260619 02:29:16.190122 23245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:29:16.190187 23245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:29:16.190260 23006 master_runner.cc:186] Error getting master registration for 127.0.98.61:39213: OK, instance_id { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" instance_seqno: 1781836155914863 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:29:16.193776 23006 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:29:16.193856 23149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a: Bootstrap starting.
I20260619 02:29:16.193871 23225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } }
I20260619 02:29:16.195202 23225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:16.195508 23199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a15e4bacead4329b6c7369517b0fb27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2f9e1f9156be4c638256be0f5f20cd46"
I20260619 02:29:16.195713 23199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:29:16.196483 23149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a: Neither blocks nor log segments found. Creating new log.
I20260619 02:29:16.197062 23199 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a15e4bacead4329b6c7369517b0fb27 in term 1.
I20260619 02:29:16.197341 23149 log.cc:824] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:29:16.197494 23023 leader_election.cc:302] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a15e4bacead4329b6c7369517b0fb27, 2f9e1f9156be4c638256be0f5f20cd46; no voters:
I20260619 02:29:16.197945 23234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:29:16.198163 23234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [term 1 LEADER]: Becoming Leader. State: Replica: 1a15e4bacead4329b6c7369517b0fb27, State: Running, Role: LEADER
I20260619 02:29:16.197739 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } } }
I20260619 02:29:16.198414 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:29:16.198508 23234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } }
I20260619 02:29:16.202003 23149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a: No bootstrap required, opened a new log
I20260619 02:29:16.202550 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.191402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.0.98.60:36839)
0619 02:29:16.191505 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.191509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.191869 (+ 360us) server_negotiation.cc:407] Connection header received
0619 02:29:16.191986 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.191989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.192038 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.192118 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.192957 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.193768 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.194919 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.195290 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.195386 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.198078 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:16.198083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:16.198089 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:16.198122 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:16.200274 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.200748 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.200752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.200754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.200833 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.201137 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.201140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.201142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.201340 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:16.201479 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:16.202288 (+ 809us) server_negotiation.cc:299] Negotiation successful
0619 02:29:16.202397 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 02:29:16.202646 23263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:29:16.202698 23263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:29:16.203169 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a15e4bacead4329b6c7369517b0fb27. Latest consensus state: current_term: 1 leader_uuid: "1a15e4bacead4329b6c7369517b0fb27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } } }
I20260619 02:29:16.203308 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:16.209933 23149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } }
I20260619 02:29:16.211058 23149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da9e9ffabe64406ac8ceccdafab4e6a, State: Initialized, Role: FOLLOWER
I20260619 02:29:16.211457 23149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } }
I20260619 02:29:16.212898 23149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:29:16.213788 23270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:29:16.209859 23157 master_runner.cc:186] Error getting master registration for 127.0.98.61:39213: OK, instance_id { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" instance_seqno: 1781836155914863 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:29:16.214377 23157 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:29:16.214762 23270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:29:16.213224 23123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a15e4bacead4329b6c7369517b0fb27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a"
I20260619 02:29:16.216063 23123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:29:16.217360 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.198881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.0.98.62:33049)
0619 02:29:16.198992 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.198996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.205713 (+ 6717us) server_negotiation.cc:407] Connection header received
0619 02:29:16.205759 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.205762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.205812 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.205897 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.207603 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.208381 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.210169 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.210511 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.210586 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.213552 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:16.213557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:16.213560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:16.213585 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:16.215278 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.215759 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.215763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.215765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.215847 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.216405 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.216409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.216411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.216627 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:16.216726 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:16.217074 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 02:29:16.217206 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:29:16.217821 23123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a15e4bacead4329b6c7369517b0fb27 in term 1.
I20260619 02:29:16.225389 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:16.199229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.0.98.61:39213)
0619 02:29:16.199333 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.199337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.206424 (+ 7087us) server_negotiation.cc:407] Connection header received
0619 02:29:16.206471 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.206474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.206521 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.206603 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.213079 (+ 6476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.213828 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.215082 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:16.215378 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:16.215466 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:16.221675 (+ 6209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:29:16.221679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:29:16.221682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:29:16.221707 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:29:16.223406 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.223874 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.223878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.223880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.223961 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:29:16.224248 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:29:16.224251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:29:16.224253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:29:16.224450 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:29:16.224562 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:16.225131 (+ 569us) server_negotiation.cc:299] Negotiation successful
0619 02:29:16.225238 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:29:16.225741 23271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } } }
I20260619 02:29:16.225888 23271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:29:16.230158 23123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a [term 1 FOLLOWER]: Refusing update from remote peer 1a15e4bacead4329b6c7369517b0fb27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:29:16.230196 23199 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46 [term 1 FOLLOWER]: Refusing update from remote peer 1a15e4bacead4329b6c7369517b0fb27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:29:16.230835 23234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:29:16.231796 23234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:29:16.237995 23271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a15e4bacead4329b6c7369517b0fb27. Latest consensus state: current_term: 1 leader_uuid: "1a15e4bacead4329b6c7369517b0fb27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } } }
I20260619 02:29:16.238195 23271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:29:16.240329 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a15e4bacead4329b6c7369517b0fb27. Latest consensus state: current_term: 1 leader_uuid: "1a15e4bacead4329b6c7369517b0fb27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } } }
I20260619 02:29:16.240491 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:29:16.242178 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a15e4bacead4329b6c7369517b0fb27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } } }
I20260619 02:29:16.242316 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:16.242677 23271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a15e4bacead4329b6c7369517b0fb27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } } }
I20260619 02:29:16.242785 23271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:29:16.244233 23284 mvcc.cc:205] Tried to move back new op lower bound from 7298400895908556800 to 7298400895792492544. Current Snapshot: MvccSnapshot[applied={T|T < 7298400895908556800}]
W20260619 02:29:16.244884 23283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:29:16.244983 23283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:29:16.246496 23234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a15e4bacead4329b6c7369517b0fb27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } } }
I20260619 02:29:16.246615 23234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27 [sys.catalog]: This master's current role is: LEADER
I20260619 02:29:16.247880 23286 mvcc.cc:205] Tried to move back new op lower bound from 7298400895908556800 to 7298400895792492544. Current Snapshot: MvccSnapshot[applied={T|T < 7298400895908556800}]
I20260619 02:29:16.248049 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a15e4bacead4329b6c7369517b0fb27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15e4bacead4329b6c7369517b0fb27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33049 } } peers { permanent_uuid: "4da9e9ffabe64406ac8ceccdafab4e6a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39213 } } peers { permanent_uuid: "2f9e1f9156be4c638256be0f5f20cd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36839 } } }
I20260619 02:29:16.248178 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:29:16.250804 23270 catalog_manager.cc:1347] Generated new cluster ID: d0db2a44989344e7b74671af2fceb6ab
I20260619 02:29:16.250872 23270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:29:16.262253 23270 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:29:16.264863 23270 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:29:16.282538 23270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a15e4bacead4329b6c7369517b0fb27: Generated new TSK 0
I20260619 02:29:16.284126 23270 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:29:16.319347 23231 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:16.315973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.0.98.60:36839)
0619 02:29:16.316227 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.316232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.318253 (+ 2021us) server_negotiation.cc:407] Connection header received
0619 02:29:16.318470 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.318474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.318527 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.318581 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.319209 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48558: BlockingRecv error: recv got EOF from 127.0.0.1:48558 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20260619 02:29:16.320711 23076 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:16.316944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.0.98.62:33049)
0619 02:29:16.317060 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.317065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.319832 (+ 2767us) server_negotiation.cc:407] Connection header received
0619 02:29:16.320003 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.320006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.320056 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.320106 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.320594 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59064: BlockingRecv error: recv got EOF from 127.0.0.1:59064 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 02:29:16.322075 23155 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:16.317642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.0.98.61:39213)
0619 02:29:16.317768 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.317773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.321168 (+ 3395us) server_negotiation.cc:407] Connection header received
0619 02:29:16.321329 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.321332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.321379 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.321441 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.321960 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53510: BlockingRecv error: recv got EOF from 127.0.0.1:53510 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 02:29:16.326318 23155 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:16.324914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.0.98.61:39213)
0619 02:29:16.325013 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.325017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.325401 (+ 384us) server_negotiation.cc:407] Connection header received
0619 02:29:16.325654 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.325657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.325730 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.325779 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.326226 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53514: BlockingRecv error: recv got EOF from 127.0.0.1:53514 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 02:29:16.331087 23155 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:16.329766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.0.98.61:39213)
0619 02:29:16.329857 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.329861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.330280 (+ 419us) server_negotiation.cc:407] Connection header received
0619 02:29:16.330457 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.330460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.330504 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.330550 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.330996 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53524: BlockingRecv error: recv got EOF from 127.0.0.1:53524 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 02:29:16.336342 23155 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:16.335045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53540 (local address 127.0.98.61:39213)
0619 02:29:16.335138 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.335142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.335563 (+ 421us) server_negotiation.cc:407] Connection header received
0619 02:29:16.335727 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.335730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.335774 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.335820 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.336254 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53540: BlockingRecv error: recv got EOF from 127.0.0.1:53540 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 02:29:16.343855 23155 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:16.342442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.0.98.61:39213)
0619 02:29:16.342568 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.342572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.343010 (+ 438us) server_negotiation.cc:407] Connection header received
0619 02:29:16.343185 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.343188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.343233 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.343280 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.343756 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53554: BlockingRecv error: recv got EOF from 127.0.0.1:53554 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 02:29:16.355629 23155 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:16.354182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.0.98.61:39213)
0619 02:29:16.354287 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.354291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.354757 (+ 466us) server_negotiation.cc:407] Connection header received
0619 02:29:16.354940 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.354943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.354993 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.355042 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.355527 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53556: BlockingRecv error: recv got EOF from 127.0.0.1:53556 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 02:29:16.375794 23155 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:16.374162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.0.98.61:39213)
0619 02:29:16.374284 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.374288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.374844 (+ 556us) server_negotiation.cc:407] Connection header received
0619 02:29:16.375044 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.375048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.375100 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.375151 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.375677 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53570: BlockingRecv error: recv got EOF from 127.0.0.1:53570 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 02:29:16.412662 23155 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:16.410797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.0.98.61:39213)
0619 02:29:16.410926 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.410932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.411584 (+ 652us) server_negotiation.cc:407] Connection header received
0619 02:29:16.411796 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.411800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.411858 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.411916 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.412540 (+ 624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53580: BlockingRecv error: recv got EOF from 127.0.0.1:53580 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 02:29:16.480494 23155 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:16.479206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.0.98.61:39213)
0619 02:29:16.479408 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.479412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.479504 (+ 92us) server_negotiation.cc:407] Connection header received
0619 02:29:16.479654 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.479656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.479706 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.479787 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.480343 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53586: BlockingRecv error: recv got EOF from 127.0.0.1:53586 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260619 02:29:16.611790 23155 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:16.610583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.0.98.61:39213)
0619 02:29:16.610713 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.610716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.610943 (+ 227us) server_negotiation.cc:407] Connection header received
0619 02:29:16.611046 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.611048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.611124 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.611246 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.611658 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53596: BlockingRecv error: recv got EOF from 127.0.0.1:53596 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 02:29:16.870995 23155 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:29:16.869987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.0.98.61:39213)
0619 02:29:16.870135 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:16.870139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:16.870157 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:29:16.870298 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:16.870300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:16.870348 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:16.870419 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:29:16.870859 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53612: BlockingRecv error: recv got EOF from 127.0.0.1:53612 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:29:17.203917 23263 catalog_manager.cc:1259] Loaded cluster ID: d0db2a44989344e7b74671af2fceb6ab
I20260619 02:29:17.204030 23263 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46: loading cluster ID for follower catalog manager: success
I20260619 02:29:17.205724 23263 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46: acquiring CA information for follower catalog manager: success
I20260619 02:29:17.206857 23263 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2f9e1f9156be4c638256be0f5f20cd46: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:29:17.246289 23283 catalog_manager.cc:1259] Loaded cluster ID: d0db2a44989344e7b74671af2fceb6ab
I20260619 02:29:17.246419 23283 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a: loading cluster ID for follower catalog manager: success
I20260619 02:29:17.248186 23283 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a: acquiring CA information for follower catalog manager: success
I20260619 02:29:17.249346 23283 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4da9e9ffabe64406ac8ceccdafab4e6a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:29:17.388926 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:17.385207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53622 (local address 127.0.98.61:39213)
0619 02:29:17.385429 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:17.385433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:17.385450 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:29:17.385548 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:17.385551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:17.385676 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:17.385824 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:29:17.386084 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:17.386632 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:17.387532 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:17.387775 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:17.387866 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:17.388152 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:29:17.388601 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:29:17.388693 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:17.388747 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 02:29:17.388798 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":67,"threads_started":1}
I20260619 02:29:17.395972 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:17.392228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.0.98.60:36839)
0619 02:29:17.392472 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:17.392475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:17.392487 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:29:17.392564 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:17.392566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:17.392679 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:17.392775 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:29:17.392989 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:17.393504 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:17.394399 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:17.394642 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:17.394734 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:17.395019 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:29:17.395557 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:29:17.395652 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:17.395707 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 02:29:17.395756 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":103,"threads_started":1}
I20260619 02:29:17.403237 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:17.399063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.0.98.60:36839)
0619 02:29:17.399167 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:17.399171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:17.399479 (+ 308us) server_negotiation.cc:407] Connection header received
0619 02:29:17.399648 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:17.399651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:17.399702 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:17.399749 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:29:17.400088 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:17.400858 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:17.402275 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:17.402506 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:17.402583 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:17.402795 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:29:17.402864 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:29:17.402944 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:17.403050 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 02:29:17.403102 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:29:17.404086 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:17.399289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.0.98.61:39213)
0619 02:29:17.399405 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:17.399410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:17.399571 (+ 161us) server_negotiation.cc:407] Connection header received
0619 02:29:17.399752 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:17.399756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:17.399806 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:17.399882 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:29:17.400194 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:17.400944 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:17.401948 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:17.402184 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:17.402294 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:17.403615 (+ 1321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:29:17.403685 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:29:17.403763 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:17.403882 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 02:29:17.403931 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:29:17.404403 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:29:17.398935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59076 (local address 127.0.98.62:33049)
0619 02:29:17.399316 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 02:29:17.399321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:29:17.399532 (+ 211us) server_negotiation.cc:407] Connection header received
0619 02:29:17.399690 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:29:17.399697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:29:17.399840 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:29:17.399989 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:29:17.400355 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:17.400940 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:17.402642 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:29:17.402980 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:29:17.403105 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:29:17.403543 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:29:17.403858 (+ 315us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:29:17.403942 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:29:17.404193 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 02:29:17.404253 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":84,"threads_started":1}
I20260619 02:29:17.407236 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 23006
I20260619 02:29:17.413996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 23081
I20260619 02:29:17.419692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjWwzrN/build/debug/bin/kudu with pid 23157
2026-06-19T02:29:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3058 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6124 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283529 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST